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user  is  assigned  a  communication  link  whose  capacity  is  large  enough  to  support 
that  user's  peak  load.  While  timing  is  easily  preserved  by  such  systems,  it 
is  difficult  to  share  communication  links.  This  research  explores  systems  in 
which  the  communication  channels  are  shared  among  many  users,  thus  causing  delay 
and  capacity  to  vary. 

The  first  area  of  concentration  is  an  examination  of  some  important  fectors 
which  adversely  affect  delay  in  packet  switched  networks.  The  ARPANET  is  used 
as  an  example  of  such  a  network.  We  focus  on  loop  control  in  adaptive  routing, 
priority  assignment,  and  the  effect  of  periodic  update  routing  in  large  net¬ 
works.  Suggestions  are  offered  for  performance  improvement  in  these  areas. 
Analysis  and  simulation  are  used  to  predict  the  magnitude  of  improvement.  By 
modifying  the  ARPANET  procedures  in  the  manner  suggested,  measurement  and  simu¬ 
lation  indicate  that  a  40%  to  50%  reduction  in  average  delay  may  be  achieved. 

Even  with  improved  performance  a  packet  switched  network  exhibits  variabld 
delay  due  in  general  to  the  possible  queueing  of  packets  at  channels  in  the 
network.  With  stream  traffic  it  is  important  to  preserve  the  relative  timing 
of  the  information  as  closely  as  possible.  This  is  accomplished  by  smoothing 
the  departure  of  information  with  buffering.  This  is  the  second  main  area  of 
concentration.  We  propose  buffering  schemes  which  adapt  to  changing  network 
delay  and  which  trade  output  smoothness  against  buffering  delay.  The  perform¬ 
ance  of  the  buffering  strategies  is  compared  by  analysis  and  simulation. 
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ABSTR(^CT 


This  research  deals  with  the  transmission  of  stream 
traffic  through  packet  switched  networks.  Stream  traffic 
communication  is  characterized  by  1)  a  requirement  of  small 
response  time  and  moderate  throughput/  the  fact  t^'at  2)  tim¬ 
ing  is  an  integral  part  of  the  information  and  that  3)  the 
information  is  redundant  and  somewhat  tolerable  to  loss. 
Remote  woice  communication  provides  an  example  of  stream 
traffic  communication.  Traditional  dedicated  communication 
systems  supporting  stream  traffic  have  exhibited  fixed  capa¬ 
city  and  fixed  delay.  In  such  systems  each  user  is  assigned 
a  communication  link  whose  capacity  is  large  enounh  to  sup¬ 
port  that  user's  peak  load.  while  timing  is  easily 
preserved  by  such  systems/  it  is  difficult  to  shsre  communi¬ 
cation  links.  This  research  explores  systems  in  which  the 
communication  channels  are  shared  among  many  users/  thus 
causing  delay  and  capacity  to  vary. 

The  first  area  of  concentration  is  an  examination  of 
some  important  factors  which  adversely  af'fect  delay  in  pack¬ 
et  switched  networks.  The  ARPANET  is  used  as  an  example  of 
such  a  network.  we  focus  on  loop  control  in  adaptive  rout- 
ino/  oriority  assionment/  and  the  effect  of  periodic  uodate 
routing  in  large  networks.  Suggestions  are  offered  for  per¬ 
formance  improvement  in  these  areas.  Analysis  and  simula¬ 
tion  are  used  to  predict  the  maonitude  of  improvement.  Pv 
modifying  the  ARPANET  orocedures  in  the  manner  suggested/ 
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■easurenent  and  siaulation  Indicate  that  a  4CX  to  50%  reduc¬ 
tion  In  average  delay  may  be  achieved. 

Even  vlth  improved  performance  a  packet  switched  net¬ 
work  exhibits  variable  delay  due  In  general  to  the  possible 
queueing  of  packets  at  channels  In  tne  network,  with  stream 
traffic  It  Is  Important  to  preserve  the  relative  timinq  of 
the  Information  as  closely  as  possible.  This  Is  accom¬ 
plished  by  smoothing  tne  departure  of  Information  with 
buffering.  This  Is  the  second  main  area  of  concentration, 
■e  propose  buffering  schemes  whicn  adapt  to  changing  network 
delay  and  which  trade  output  smoothness  against  buffering 
delay.  The  performance  of  the  buffering  strategies  Is  com¬ 
pared  by  analysis  and  simulation. 
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CHAPTER  1 


INTRODUCTION 


1.1  Resent  histatx  of  eaitsi  saiishioa 

i;ince  the  development  of  the  ARPANET  fRobe  703  in  the 
late  1960s  by  the  Advanced  Research  Projects  Agency  of  the 
United  States  Department  of  Defense^  there  has  been  ar  ever 
increasino  amount  of  activity  in  the  area  of  packet  switch¬ 
ing  research^  development/  and  implementation.  The  ARPANET/ 
built  primarily  by  Bolt/  Eeranek  and  Newman/  Inc-/  Cam¬ 
bridge/  Massachusetts  (BBN)/  is  the  most  widely  known  exam¬ 
ple  of  a  packet  switched  network.  Other  networks  are  emerg¬ 
ing  as  many  agencies/  countries  and  federations  are  current¬ 
ly  funding  research  and/or  building  packet  switched  net¬ 
works.  Among  these  are  WWMCCS  CBeno  713/  NPL  [Barb  703/ 
CYCLADES  CPour  733/  DATAPAC  COPAC  743/  C0ST11  [Barb  723/  to 
name  a  few.  T^e  first  commercial  oacket  switching  service 
is  now  in  operation  in  the  U.  S.  (operated  by  Telenet  Com¬ 
munications  Corporation)  C'*ath  753  . 

Extensions  to  the  ARPANET  form  of  packet  switching  have 
taken  place  and  have  lead  to  experimental  systems  --  ALOHA 
CMOra  703/  PRNET  CKahn  753/  SATNET  CKlei  733/  ETHERNET 
CHetc  773.  Currently/  the  interconnection  of  surh  networks 
[McKe  74a3  and  the  standardization  of  protocols  CPouz  753/ 
CHove  763  are  each  of  considerable  interest  in  the  data 
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communications  community.  Also  of  interest  are  aspects  of 
secure  communications  CFarb  7')3. 

Iti£  fgilyrgs  of  Qacjcgi  sttiubioa 

Here  we  describe  the  important  (so  far  as  we  are  con¬ 
cerned  here)  properties  of  a  packet  switched  network.  ft 
packet  switched  network  (of  the  ARPANET  form)  incorporates 
switching  computers  within  the  communications  media.  tfie 
network  consists  of  HOSTs  (i.e.#  packet  or  message  sources 
--  computers  or  terminals)  and  a  subnet  to  which  the  HOSTs 
are  attached.  The  supnet  contains  switching  nodes  (called 
IMPS  in  the  ARPANET  CHear  7CD)  which  are  connected^  with 
some  topology/  by  a  set  of  communications  lines  or  channels. 
HOSTs  are  connected  to  the  network  at  switching  nodes.  » 
source  HOST  sends  a  message  (to  a  destination  HOST)  hy 
delivering  the  message/  with  the  address  of  the  destination 
HOST/  to  its  connected  switching  node.  The  switching  node 
then  creaks  the  message  into  one  or  more  packets.  Each 
packet  is  then  forwarded  (if  necessary)  by  the  switching 
node  to  one  of  its  neighbors  (i.e./  another  switching  node 
which  is  directly  connected  to  the  forwardino  node)  the 
choice  of  neighbors  being  governed  by  the  routing  procedure. 
Each  node/  which  encounters  the  packet/  forwards  it  to  a 
neighbor  the  choice  of  which  is  again  based  on  the  routing 
p rocedure  . 

Often  there  is  some  form  of  error  detection  and 
recovery  in  the  forwarding  process  in  order  to  assure  the 
correct  receipt  of  packets.  In  the  ARPANET/  for  example/ 
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there  is  a  checksum  appended  to  the  packet  at  the  sender 
which  is  checked  by  the  receiver.  Only  when  the  checksum  is 
correct  and  buffer  space  is  available  does  the  receiver 
return  (sometimes  "piggybacked"  on  reverse  traffic)  an  ack¬ 
nowledgement  (ACK)  to  the  sender.  If  a  sent  packet  has  not 
received  its  ACK  within  a  certain  time  (currently  128  mil¬ 
liseconds)/  then  the  packet  is  retransmitted. 

The  forwarding  process  is  repeated  until  the  packet 
arrives  at  a  switching  node  which  is  connected  to  the  desti¬ 
nation  HOST/  at  which  ooint  the  packet  is  reassembled  with 
the  other  packets  of  the  same  message.  When  all  its  packets 
have  arrived  the  message  is  sent  to  the  destination  HOST. 
In  the  ARPANET  this  final  step  creates  an  end-to-end  ack¬ 
nowledgement  called  a  Ready-for-Next-Message  (RFN»")  which  is 
sent  back  to  the  sender. 

At  each  forwarding  step/  often  called  a  hop/  along  the 
way  a  packet  incurs  some  processing  delay  which  is  required 
to  make  the  routing  decision  (i.e./  to  which  neighbor  to 
route  this  packet).  Also/  it  may  encounter  a  cueue  of  pack¬ 
ets  waiting  to  be  sent  to  the  neighbor  and  must  therefore 
wait  until  all  (higher  or  equal  priority)  packet®  have  been 
served  (i.e./  sent  to  the  neighbor)  before  it  may  use  the 
channel.  In  particular/  this  may  result  in  variable  delay. 
That  is/  a  message  sent  from  A  to  F-  at  time  t'T  may  experi¬ 
ence  a  different  delay  than  a  similar  message  sent  at  time 
t1.  The  maximum  attainable  throughput  from  »  to  r  may  vary 
with  time  as  well/  since  it  will  depend  on  the  level  of 
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interference  from  other  communicating  HOST  pairs. 

Packet  switching  promises  greater  efficiency  of 
resources  by  sharing  those  resources  amono  many  users/  f'ach 
of  which  uses  only  a  small  portion  of  the  total  capacity. 
The  savings/  at  least  in  part/  are  passed  along  to  the  user. 
With  the  rapid  growth  of  data  common i c a t i ons  we  shall  surely 
see  more  and  bigger  packet  switched  networks  emeroe. 

Pecause  of  its  digital  nature/  today’s  packet  switching 
technology  has  been  designed  and  used  chiefly  for  data  com¬ 
munications  (e.g./  t e r m i na t - t o- c ompu t er  ^and  comcuter-to- 
computer) .  Other  forms  of  common i c a t i on  (i.e./  voice)/ 
which  have  traditionally  been  accomplished  by  analog 
methods/  are  now  beginning  to  use  digital  technolooy. 

yeiifi  goes  diaixai 

The  advent  of  inexpensive/  highly  reliable  digital 
transmission  equipment  has  already  led  to  the  use  of  such 
circuits  in  "short-haul"  voice  communications  [Jame  72D. 
Also  some  AC  million  circuit  miles  of  digital  trunks  were  in 
place  by  the  end  of  1975  CTalk  773.  There  seems  little 
ooubt  that  the  use  of  digital  transmission  will  increase  as 
time  progresses.  we  are  told  by  Gallager  CGall  77a]  that 
the  military  will  use  an  digital  telephone  r.etwcrk  in 
the  198G's.  Cost  will  be  the  most  important  reason  for 
conversion  to  digital  transmission.  Accordino  to  Falk 
CFalx  773  "Bell  Canada  plans  to  save  SAC  million  a  year  in 
capital  costs  by  using  new  digital  equipment".  Another 


consideration  is  that  a  diqital  siqnal  is  much  easier  to 
encrypt  than  is  an  analog  siqnal.  This  could  lead  to  a  far 
more  secure  means  of  voice  communication. 

Simple  digitization  requires  a  data  rate  of  between  50 
and  60  kilobits  per  second  (kbps)  to  provide  quality  equal 
to  conventional  telephone  equipment-  This  is  extremely  high 
compared  to  current  requirements  in  data  communications. 
Fortunately^  speech  is  highly  redundant  and  can  therefore  be 
significantly  compressed. 

1.3  ScggjEt  £S!DE£££ii2Q  £££22L£h 

There  are  two  methods  of  compression  currently  under 
study  by  the  ARPA  Network  Speech  Compression  (NSC)  group  - 
(a)  continuous  variable  slope  delta  modulation  (CVSD) 
CForo  74]  and  (b)  linear  predictive  coding  (LPC)  CAtal  71]. 
Experimental  studies  of  remote  voice  communication  using 
these  two  schemes  have  already  taken  place  in  the  ARPANET. 
The  CVSO  scheme  transmits  one  bit  per  sample#  whose  value  (1 
or  C)  depends  on  whether  the  last  synthesized  point  is  above 
or  below  the  current  input.  This  scheme  provides  acceptable 
quality  in  the  d  to  20  kbps  reckon  [Forg  743.  The  experi¬ 
ments  in  the  ARPANET  with  this  technique  used  a  peak  rate  of 
Id  (cHps  and  experienced  delays  on  the  order  of  ?  to  4 
seconds  between  speaker  and  listener  (over  a  in  hop  network 
path)#  as  reported  by  Cohen  in  CNayl  743].  The  LPC  method 
is  based  on  a  model  of  the  vocal  tract  and  assumes  that  a 
speech  sample  may  be  approximated  by  a  linear  combination  of 


the  previous  n  samples.  The  coefficients  in  the  linear  com¬ 
bination  are  slowly  varying  quantities.  This  algorithm  pro¬ 
vides  high  quality  speech  in  the  range  of  2.4  to  9.6  kbps 
CAtal  713.  With  refinements  such  as  DELCO  CMaqi  733  /^  one 
can  achieve  average  data  rates  of  1.2  to  4.8  kbps.  A  recent 
experiment  in  the  ARPANET  had  a  peak  date  rate  of  4.1  kbps 
and  an  average  of  1.4  kbps/  with  acceptable  quality/^  using 
the  LPC  compression  scheme  CMcCa  753. 

This  research  has  created/  in  essence/  a  new  form  of 
"data"  communication/  and  provides  the  opportunity  to  apply 
Oocket  switching  technology  to  the  area  of  remote  voice  com¬ 
munication.  Human  speech  contains  a  great  deal  of  redundan¬ 
cy  and  silence.  During  silence/  with  most  dedicated  or 
switched  circuit  systems  the  channel  remains  unused,  racket 
switching  would  allow  this  dead  time  to  be  used  by  others 
thus  allowing  greater  channel  efficiency  which  eventually 
translates  to  cost  savings. 

1-^  lbs  shg£i££££lsti££  Qf  st£eam  t£slil£ 

This  dissertation  investigates  a  set  of  problems  relat¬ 
ed  to  the  use  of  packet  switching  technology  for  the  purpose 
of  stream  traffic  communication.  Remote  voice  communication 
is  ''n  example  of  stream  traffic  communication.  Let  us  de¬ 
fine  stream  traffic.  Stream  traffic  is  characterized  by  the 
following  three  properties: 

1)  small  response  time  and  moderate  throughput  are 
r equi red/ 
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2)  tiffing  is  an  integral  part  of  the  information/  and 

5)  the  information  contains  redundancy. 

F^roperty  1  allows  for  the  possibility  of  real-time 
interactive  communication  between  two  or  more  locations. 
This  property  alone  makes  stream  traffic  distinguishable 
from  the  two  classical  forms  of  data  communication.  racket 
switched  networks  have/  in  generals  been  designed  to  carry 
traffic  which  has  traditionally  been  classified  into  two 
categories:  ta)  LO  -  low  delay  (interactive)/  and  (b)  ht 
high  throughput  (file  transfer).  As  noted  by  Cohen/  Cpder- 
Leck  and  Kleinrock  COpde  743/  stream  traffic  communication 
falls  into  yet  another  category  (c)  ST  -  stream  traffic/ 
requiring  both  low  delay  and  moderate  throughput. 

Not  only  are  the  transmission  requirements  of  stream 
traffic  unique/  but  the  inform at  ion  itself  is  of  a  somewhat 
different  nature  than  the  usual  data  communication.  Proper¬ 
ty  2  indicates  that  each  unit  (bit/  if  you  will)  of  informa¬ 
tion  has  an  associated  (possibly  implied)  time  stamp  and 
that  the  relative  timing  of  the  information  should  be 
preserved  as  well  as  possible  by  the  transmission  media. 
Although  sequencino  is  important/  in  ordinary  data  communi¬ 
cations  there  is  no  notion  of  timing  associated  with  the 
information.  Unlike  traditional  data  communi ce ti on  /  the 
information  in  stream  traffic  is  somewhat  redundant.  TPis 
means  that  the  information  is  less  vulnerable  to  loss  within 
the  system  than  for  traditional  data  communication.  The 
communication  media  may  lose  a  small  fraction  of  the 
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information  without  seriously  affecting  the  quality.  This 
is  clearly  distinct  from^  say/  remote  job  entry  in  which  a 
very  precise  specification  in  the  form  of  a  prooram  is  the 
object  of  communication/  and  therefore  no  loss  at  all  can  be 
tolerated . 

This  characterization  of  stream  traffic  is  intended  to 
fit  two-  (or  more)  way  interactive  voice  communication. 
There  are  other  areas  of  communication  which  may  possess 
these  properties  as  well.  Distribution  and  local  broadcast 
of  live  (or  delayed  and  transmitted  from  the  source  in 
real-time)  radio  or  television  programming  would  appear  to 
have  the  three  properties.  While  there  is  no  interaction 
required  of  such  one-way  systems/  low  delay  may  be  required 
due  to  lack  of  buffer  space  at  the  receiver.  Television  in 
particular  requires  extremely  high  throughput  by  ARPANET 
standards.  Therefore  the  capacity  of  a  network  designed  to 
carry  television  must  have  a  capacity  so  that  the  television 
transmission  consumes  only  a  moderate  portion  of  that  capa¬ 
city. 

1*5  Ss.at£!ii£Qt  of  the  ErgbiSID 

There  are  two  main  areas  of  emphasis  which  are  explored 
in  this  dissertation. 

1)  Identify  and  examine  those  design  considerations  <ot 
packet  switched  networks)  which  have  a  significant 
impact  on  the  performance  of  stream  traffic  communi¬ 
cation  in  a  packet  switched  environment. 


2)  Given  a  (packet  switched)  networks  examine  some 
sendinq  and  receivinq  policies  which  attempt  to  bal¬ 
ance  the  preservation  of  the  relative  timing  aoainst 
end-to-end  delay. 

Since  the  ASPANFT  is  a  convenient  example  of  a  packet 
switched  network/  we  examine  it  as  a  case  study  and  attempt 
to  extract  some  general  results  about  designing  a  packet 
switched  network  aimed  at  the  ST  class/  but  still  retaining 
capabilities  in  the  other  classes.  The  emphasis  here  is  to 
reduce  delay  in  order  to  satisfy  property  (1).  loop  con¬ 
trol/  system  priorities/  and  the  effects  of  oeriodic  routinq 
updates  are  among  the  issues  of  concern  under  area  (1). 

1.6  Symmar^  of  resui£§ 

In  Chatter  2  we  investigate  the  occurrence  of  loops 
caused  by  routing  update  procedures.  “iy  routing  we  mean  the 
process  by  which  packets  are  directed  from  switch  to  switch 
throuon  the  network.  This  definition  is  intended  to  include 
the  structure  and  use  of  local  routinq  information  as  well 
as  the  updating  of  this  local  information.  Routing  algo¬ 
rithms  are  broken  into  four  classes  in  CCrow  75D/*  (a)  "non- 
adaptive"  or  fixed  -  where  the  route  between  any  two  nodes 
remains  fixed.  (b)  "Centrali?ed  adaptive"  -  in  which  routes 
are  dynamically  modified  by  a  central  overseer.  (c)  "Iso¬ 
lated  adaptive"  -  where  routes  are  changed  dynamically  by 
each  node  without  sharina  information  among  the  nodes.  (d) 


9 


“Distributed  adaptive"  -  in  which  routes  are  dynamically 
changed  by  each  node  with  information  shared  amona  the  nodes 
in  the  form  of  routing  update  packets.  It  should  be  clear 
that  loops  would  be  forbidden  in  types  (a)  and  (b)  by  a 
small  amount  of  care.  It  would  appear  to  be  impossible  to 
prevent  loops  in  type  (c).  We  therefore  concentrate  our 
efforts  in  studying  the  distributed  adaptive  type  algorithm. 
The  existence  of  routing  loops  under  a  routing  procedure  is 
a  possible  source  of  performance  degradation.  Packets  which 
are  trapped  in  loops  have  increased  network  delay.  If  a 
loop  persists  sufficiently  long#  then  interaction  is  ham¬ 
pered  if  not  destroyed  (i.e.#  property  (1)  is  violated).  In 
CiNayl  75D  a  loop-free  routing  algorithm  is  presented.  It 
was  shown  that  this  algorithm  cannot  create  loops#  tut  its 
operational  characteristics  reauired  further  examination. 
We  investigate  the  operational  characteristics  of  this  and 
other  algorithms  through  the  use  of  simulation.  A  local 
loop-free  (or  ping-pong-free)  algorithm  is  found  to  perform 
best  amona  those  algorithms  tested. 

In  ChaBlg£  3  we  consider  priority  assignment  among 
tasks  within  the  system.  An  important  issue  in  the  perfor¬ 
mance  of  a  network  is  the  assignment  of  the  priority  among 
the  various  functions  within  a  node  as  well  as  the  priori¬ 
ties  assigned  to  the  transmission  of  packets  on  the  chan¬ 
nels.  As  an  example#  we  have  found  that  under  the  current 
ARPANET  strategy#  packets  waited  as  long#  on  the  average# 
for  the  processor  as  for  the  transmission  channels.  This  is 
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due  to  the  fact  that  routing  update  packets  and  non-routing 
packets  (i.e./  data  and  control  distinct  from  routing  update 
packets)  are  processed  in  f i rst-come-f i rst-served  (FCFS) 
order  within  the  nodes.  In  addition  some  of  the  nodes  in 
the  network  spend  more  than  SOX  of  their  total  capacity  just 
processing  routing  updates/  compared  to  "only"  IPX  routing 
overhead  on  the  channels.  Examination  of  this  problem  iso¬ 
lated  to  a  single  node  suggests  that  since/  in  the  current 
ARPANET/  routing  packets  require  approximately  AT  times  the 
processing  time  that  non-rout inq  packets  require/  the  pro¬ 
cessing  of  routing  packets  should  be  done  at  a  lower  rriori- 
ty.  Simulation  of  the  system  as  a  whole/  suggests  that  the 
local  optimum  leads  to  a  global  optimum  in  terms  of  mean 
delay  and  network  throughput.  Therefore/  by  modifying  the 
priority  structure  one  could  reduce  delay  and  thereby  assist 
in  complying  with  nroperty  (1>. 

In  Chagi^r  ^  we  show  some  interesting  delay  behavior 
for  a  periodic  stream  traffic  source.  The  behavior  is  due 
to  the  periodic  nature  of  the  routing  update  procedure.  The 
current  ARPANET  scheme  provides  for  periodic  updates/  and 
with  the  growing  size  of  the  network  the  rate  of  routing 
updates  has  been  increased  to  allow  For  better  propagetion 
of  information  through  the  network  when  congestion  or 
failure  occurs.  We  show  that  one  must  pay  a  high  price/  in 
terms  of  delay/  for  this  feature  during  normal  operation 
however.  Under  normal  conditions  routing  updates  are  not 
required  very  often  since  most  routes  need  not  change  much. 
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Long  tern  data  on  ARPANET  reliability  shows  that  network 
coaponent  failures  occur  very  infrequently  compared  to  the 
routing  update  period.  These  results  suggest  an  asynchro¬ 
nous  method  of  updating  routing  tables.  This  would  result 
in  lower  average  delay  and  thus  help  provide  property  (1). 

Under  the  second  area  of  emphasis  we  concentrate  on 
methods  which  attempt  to  preserve  the  relative  timing  of  the 
information  without  destroying  property  (1).  In  Chgtt££  5 
we  examine  buffering  strategies^  at  source  and  destination^ 
which  attempt  to  minimize  gaps  in  the  output  stream  and  at 
the  same  time  attempt  to  minimize  the  delay  between  speaker 
and  listener.  We  found  that  delay  varies  quite  rapidly  and 
therefore  delay  prediction  could  not  be  used  to  adjust  to 
the  network  dynamics  on  a  message  by  message  basis#  rather 
we  found  that  the  sending  strategy  should  remain  fixed  for 
"long"  periods. 

More  extensive  results  are  found  to  predict  the  perfor¬ 
mance  o'^  destination  buffering  schemes.  We  define  some 
delay  prediction  techniques  and  two  playout  methods.  Based 
on  some  assumptions  on  system  delay  we  have  developed  models 
of  the  system  behavior  in  terms  of  delay  and  gap  probabili¬ 
ty.  The  solution  of  the  models  requires  the  knowledge  of 
the  system  delay  distribution.  Fxact  results  are  obtained 
for  the  exponential  distribution  and  shifted  exponential 
distribution.  Numerical  integration  is  used  to  obtain 
results  for  an  Erlang  family  of  distributions  which  previous 
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Models  CKlel  64]  tell  us  is  a  sonewhst  better  nodel  of  net¬ 


work  delay.  Finally  by  simulation  we  compare  the  perfor¬ 
mance  of  our  delay  variation  estimation  technique  to  a  delay 
tracker  method  which  has  been  used  experimentally  in  the 
ARPANET/  and  to  a  planned  revision  of  that  scheme. 

£bSEi££  ^  lists  our  conclusions  and  sugoests  some  areas 
of  interest  for  further  research. 

Gur  critical  examination  of  ARPANET  procedures  has  had 
an  impact  not  only  stream  traffic  communication/  but  on  the 
efficiency  of  packet  switching  in  general.  The  study  of  the 
performance  of  destination  buffering  schemes  has  provided  a 
framework  in  which  other  such  techniques  may  be  examined. 
We  believe  strongly  that  stream  traffic  communication  (re¬ 
mote  voice  communication  in  particular)  is  within  the  realm 
of  uses  for  packet  switched  networks/  but  much  work  is  need¬ 
ed  in  order  to  produce  a  usable  system. 
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CHAPTER  2 


LOOP  CONTROL  IN  ADAPTIVE  ROUTING* 


2-1  lDL£S^U£ll&Q 

In  a  packet-switched  network  in  which  some  scheme  of 
adaptive  routing  of  packets  is  usedr  there  exists  the  possi¬ 
bility  that  packets  will  become  trapped  in  loops.  That  is^ 
packets  may  be  routed  in  such  a  way  that  they  return  several 
times  to  some  set  of  nodes  for  at  least  a  finite  period  of 
time  before  reaching  their  eventual  destination^  thus  wast¬ 
ing  network  bandwidth  and  significantly  increasing  message 
delay.  Routing  loops  are  of  concern  in  stream  communication 
primarily  because  of  this  increased  delay  effect.  In  this 
chapter  we  consider  procedures  for  controlling  such  loops. 

The  problem  of  looping  in  adaptive  routing  has  been 
known  to  exist  for  some  time  CKahn  71]^  CFult  7?lr 
CGerl  733#  CMcQu  743#  CCeqr  753#  CPick  763#  CGell  773.  Pre¬ 
vious  approaches  to  this  problem  have  been  to  detect  and 
remove  loops  CKahn  713#  CFult  723#  or  to  reduce  the  in¬ 
cidence  of  such  loops  ChicQu  743#  CCegr  753  .  Gerla  CGerl  733 
proved  that  an  optimal  routing  policy  must  be  loop-free. 
More  recently#  with  a  procedure  known  as  the  "last  m  nodes 
visited"  (LMNV)  algorithm  CPick  763#  the  packets  are 


•  This  chapter  is  a  revised  version  of  CNayl  753. 
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prevented  from  loopinp  in  loops  of  size  m  nodes  or  smaller. 
GalLager  has  created  an  optimal  routing  scheme  which  is 
loop-free  C6all  773.  We  first  give  two  examples  of  existing 
networks  whose  adaptive  routing  algorithms  can  cause  such 
loops,  we  next  investigate  the  order  of  magnitude  of  the 
degradation  due  to  loops.  We  then  describe  a  new  ping- 
pong-free  and  a  new  loop-free  algorithm.  A  formalism  for 
dealing  with  the  loop  problem  is  introduced  in  order  to 
prove  the  loop-free  properties  of  the  algorithms.  Then  by 
way  of  simulation  we  compare  the  performance  of  several  of 
the  routing  update  schemes. 

Z.l  Exascigs  Ql  iSSCiOS 

Below  we  describe  four  routing  update  procedures  at 
least  three  of  which  may  cause  loo os  to  occur.  There  is  a 
degree  of  commonality  among  these  schemes.  In  each  oro- 
cedure  there  exists  a  routing  table  at  each  node  whose  en¬ 
tries  indicate  the  direction  (i.e.^  the  channel  or  neighbor 
address)  in  which  to  send  a  packet  headed  for  a  particular 
destination  (see  Figure  2.1).  When  a  packet  arrives  at  a 
node  (say  node  i)/  one  uses  that  packet's  destination  ad¬ 
dress  d  as  an  index  into  the  routing  table  to  determine  the 
channel  c(i»d)  over  which  to  route  that  packet.  The  differ¬ 
ences  in  these  routing  procedures  appear  in  the  updating  of 
these  tables. 
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2.2.1  AEEiNEI  Q£&££dU£il 

Each  of  the  first  three  procedures  have  been  used  in 
the  ARPANET  CRobe  7C]»  CMcQu  72D.  The  first  two  can  cause 
loops.  The  third  scheme  is  currently  in  operation  and  is 
believed  to  prevent  at  least  ping-pong  loops. 

2. 2. 1.1  Lo2B  BEone  routing  ILPR) 

In  the  ARPANET  each  entry  in  the  routing  table  consists 
of  (among  other  things)  a  delay  estimate  and  the  address  of 
the  channel  for  which  that  delay  estimate  holds  CMcUu  743» 
CMcGu  74a]^  CKlei  761.  The  table  is  updated  upon  the  ar¬ 
rival  of  a  routing  message  from  a  node’s  neighbor.  The 
routing  message  is  a  copy  of  the  delay  portion  of  the 
neighbor's  routing  table.  The  delay  estimates  ir  that 
node's  routing  table  are  compared  with  the  routing  table 
delay  estimate  entries  in  the  arriving  update  message  plus 
the  delay  from  that  node  to  its  neighbor  (currently  4  units 
+  1  unit  for  each  packet  on  that  channel's  queue).  The 

smaller  of  the  two  values  replaces  the  routing  table  esti¬ 
mate  and  the  "best  delay  channel"  is  changed  if  necessary, 
here  precisely/  Node  i  upon  receipt  of  a  routing  update 
packet  from  neighbor  node  j  performs  the  followinn  algorithm 
for  each  d: 

LEE 

1  t'(i/d)  =  t(j/d)  ♦  q(i/j)  ♦  h 

2  if  t'(i/d)  <  t(i/d)  then  c(i/d)  =  c'(i/i) 

3  if  c(i/d)  =  c'(i/j)  then  t(i/d)  =  t’(i/d) 
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p 


where  t(j/d)  =  the  delay  estimate  from  node  j  to  node  df 
q(i^j)  =  the  length  (in  packets)  of  the  queue  from  node  i  to 
neighbor  node  h  =  the  one  hop  delay  bias  (currently  A 
units  in  the  ARPANET)#  c(i#d)  =  the  (next)  channel  on  which 
to  route  a  packet  residing  at  node  i  destined  for  node  d# 
and  cMi/-])  is  the  channel  which  connects  neighbors  i  and  j. 
This  is  a  simplification  of  the  actual  algorithm#  but  the 
essential  features  are  retained. 

Consider  the  network  topology  pictured  in  Figure  2.2. 
we  are  concerned  with  the  estimated  delay  from  nodes  P  and  C 
to  some  distant  node  D.  We  will  use  the  following  notation 
of  IKlei  7ba): 

d/A  d  =  estimated  delay  to  node  D 
A  =  next  neighbor  in  path  to  D 

B->C  node  9  sends  a  routing  update  to  C 


Figure  2.2.  A  portion  of  a  network. 

The  following  sequence  may  occur  if  B's  delay  estimate  to  D 
increases  rapidly  (at  the  second  step  in  the  sequence). 


Node  P 

Node  C 

A 

3 

t/A 

t+4/B 

A->B 

t+9/ A 

C->B 

t+8/C 

B  sends  an  update  to  C  which  sets  up  the  initial  state. 
Non-empty  queues  are  formed  within  A  and  B  (B  in  A's  direc¬ 
tion  and  ^  away  from  B)  when  A  sends  an  update  to  B.  This 
increases  B's  delay  estimate  by  9  units.  C  then  instructs  P 
(based  on  fa's  old  delay  estimate)  that  the  delay  via  C  is 
now  siraller.  This  causes  0  to  point  the  route  toward  but 
C  still  points  toward  P.  Therefore  a  loop  is  created. 

2.2.1.<;  Hgi^-down  rgyiing  <HDR) 

This  kind  of  loop  was  eliminated  by  the  addition  of  the 
constraint  that  if  the  delay  estimate  changes  by  more  than  ** 
units  on  a  given  line  between  updates  then  the  node  enters 
"hold-down"  state  on  that  line  and  will  not  switch  (even 
though  the  delay  estimates  may  be  better  in  another  direc¬ 
tion)  for  approximately  two  seconds.  We  will  not  attempt  to 
explain  this  further  than  to  say  that  it  allows  for  news  of 
drastic  changes  to  propagate  through  a  part  of  the  network 
before  routing  changes  are  made.  For  further  details  see 
Ct^cWu  7A]. 

Let  us  consider  what  may  happen  when  P's  delay  estimate 
to  node  D  gradually  increases,  we  assume  that  there  are 
twice  as  many  routing  messaaes  from  C  to  0  as  from  R  to  C. 
This  is  possible  because  the  rate  at  which  updates  occur  is 
based  on  line  speed  and  line  utilization.  For  a  50  kbps 
line  they  occur  at  a  minimum  rate  of  one  every  640  msec,  and 
a  maximum  of  one  every  1 ?8  msec.  (See  Chapter  4  for  a  more 
complete  description.)  We  denote  line  hold-down  by  an 
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exclamation  point  in  the  followino  sequence. 


Node  0 

Node  C 

B->C 

t/A 

t+4/0 

A->B 

t+5/A 

1 

V 

CD 

(no  change) 

A->B 

t+9/ A 

C->B 

t+8/C 

B->C 

t  +  13/ 

CD 

A 

1 

t+17/C' 

T^is  loop  (called  a  looo-trao  in  CKlei7533)  will  remain  for 
a  period  of  approximately  two  seconds  (the  hold-down  time)! 
Conditions  similar  to  this  have  been  observed  on  several 
occasions/  and  account/  at  least  in  part/  for  the  degrada¬ 
tions  suffered  during  experiments  with  stream  communication 
in  the  ARPANET  CCohe  7A]/  CNayl  743/  CForg  753. 

2. 2. 1.3  (Sadifiad  heid'dano  Eeatino  (khbe) 

Notice  that  the  example  of  a  loop  under  HDP  is  a  result 
of  the  variable  rate  updatinn.  After  loop-trars  were  ob¬ 
served  CNayl  753/  the  hold-down  scheme  was  modified  in  1')75 
to  make  the  criteria  for  entering  hold-down  independent  of 
update  rate.  That  is  to  say  whenever  a  delay  estimate 
changes  by  more  than  eight  units  over  an  arbitrary  number  of 
updates  then  hold-down  is  entered.  This  means  that  a  gradu¬ 
al  build  up  of  traffic  can  cause  hold-down  to  occur.  In 
fact  hold-down  would  occur  prior  to  many  routine  changes. 


This  change  may  have  produced  a  loop-free  routing  algorithm. 
(We  shall  not  attempt  a  proof  or  counterexample  here.) 
Surely  some  (possibly  many)  desirable  route  changes  are 
eliminated  or  delayed  by  such  a  scheme  however. 

2.2.2  The  IlfiAJ  NET  pro££dure 

This  loop  phenomenon  is  not  unique  to  the  ARPANET/  as 
we  now  show.  The  entries  in  the  routing  table  proposed  for 
the  TIDAS  network  in  CCeqr  753  contain  a  delay  estimate  for 
each  of  the  node's  channels  to  a  destination.  The  early 
ARPANET  had  such  tables  as  well  until  the  size  of  the  net¬ 
work  prohibited  them.  when  a  packet  arrives/  the  routing 
procedure  chooses  the  channel  with  the  smallest  delay  esti¬ 
mate  (to  the  packet’s  destination)  in  the  table/  ana  routes 
the  packet  over  that  channel.  Updates  of  the  table  are  done 
in  two  ways: 

a)  "inside"  (i.e./  using  only  local  information:  queue 

lengths  and  previous  delay  estimates). 

b)  "outside"  (i.e./  using  information  contained  in  a  rout¬ 

ing  message  from  one's  neighbor). 

The  routing  message  contains  a  delay  estimate  vector  which 
gives  the  delays  from  the  sending  neighbor  but  not  through 
the  receiver  of  the  routing  message.  That  is/  when  A  sends 
a  routing  message  to  3/  the  second  best  delay  estimate  is 
sent  for  those  destinations  whose  best  oelay  channel  is 
toward  9.  This  method  is  referred  to  as  "split  horizon" 
updating  in  CCegr  753.  We  shall  use  a  t wo-component  notation 
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siailsr  to  the  above  with  the  network  structure  shown  in 
Figure  2,2.  The  following  may  occur  with  the  stated  algo¬ 
rithm. 

Node  Node  C 

t/A,  t+12yc  t+2/B,  t+10/E 

A->B  t+13/A,  t+12/C  t+2/B,  t+10/F 

This  loop  may  be  eliminated  with  the  next  routing  message^ 
or  it  may  set  up  a  chain  of  loops  in  the  direction  of  E.  We 
are  merely  pointing  out  here  that  loops  form  easily  under 
this  routing  procedure.  In  fact^  in  simulations  of  this 
procedure  "ping  ponging"  was  clearly  present.  According  to 
Cegrell  CCegr  75]  ping-ponging  was  significantly  reduced  by 
the  split  horizon  technioue  and  immediate  updating  in  the 
case  of  node  or  line  failures. 

Although  loops  involvino  more  than  two  nodes  are  possi¬ 
ble  with  both  of  the  above  schemes^  we  will  not  present  anv 
such  examples.  Nulti-node  loops  cause  a  much  larger  degra¬ 
dation  in  network  performance  than  do  2-node  (pine-pong) 
loops/  but  their  likelihood  of  occurrence  appears  to  be  sia- 
nificantly  smaller  than  that  of  the  two-node  loop.  It  ap¬ 
pears  to  be  impossible  to  prevent  these  loops  when  each  node 
has  only  local  information.  The  feasibility  of  each  node 
carrying  and  adequately  updating  global  information  is  ques- 
t ionaole. 

In  the  next  section  we  examine  the  magnitude  of  the 
degradation  in  the  ARPANET  caused  by  the  occurrence  of 
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loops . 


2.3  2eaE3£t2li2C  dye  ts  Loops 

Havina  shown  that  loops  can  occuiv  we  now  examine  the 
degradation  to  network  performance  caused  by  loops  in  terms 
of  the  length  of  time  they  naturally  persist.  That  is  to 
sayr  we  wish  to  determine  how  long  it  takes  to  eliminate  a 
loop  under  the  current  ARPANET  routing  update  rates.  This 
persistence  time  is  of  interest  since^  if  loops  disappear 
quicklyr  then  there  would  be  no  need  for  prevention  or  rapid 
detection  and  removal.  We  first  define  the  following  quan¬ 
tities. 

p  =  Routing  update  period, 
n  =  Number  of  nodes  in  the  loop. 

I  =  Total  local  delay  estimate  in  the  loop  (in  the 
current  implementation  I  =  A*n^  assuming  the  ab¬ 
sence  of  packets  traversing  the  loop) 
d  =  minimum  delay  estimate  difference  over  all  neigh¬ 
bors  outside  the  loop, 
k  =  Processing  time  for  a  routing  message. 

X  =  Transmit  and  propagation  delay  for  a  routing  mes¬ 
sage. 

y  =  k  ♦  X . 

The  loop  will  be  cleared  whenever  a  routing  update/- 
from  outside  the  loop/  arrives  and  results  in  a  smaller 
delay  estimate  directing  packets  out  of  the  loop.  This  can 
happen  as  quickly  as  it  takes  to  process  an  arriving  routing 
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message/  if  delay  estimates  outside  the  loop  decrease  Quick¬ 
ly,  Thus  the  minimum  breakino  time  is  essentially  zero.  If 
we  assume  that  tne  delay  estimates  outside  the  loop  remain 
fixed/  then  we  must  traverse  the  looo  with  routing  updates  a 
maximum  of  ceil<d/l)  times  (where  ceil  is  defined  to  be  the 
usual  ceiling  function/  i.e./  ceil(A)  =  the  smallest  integer 
greater  than  or  equal  to  A).  This  causes  the  internal  delay 
estimate  to  be  greater  than  the  external  delay  estimate. 
The  maximum  time  which  a  loop  can  persist/  assuming  the 
exterior  delay  remains  fixed/  is 

Cl  +  y  ♦  n(p  +  y)ceiltd/l) 

To  compute  a  minimum  we  assume  that  the  nodes  are  synchron¬ 
ized  in  such  a  way  that  a  routing  message  is  sent  just  after 
the  receipt  of  one  from  a  neighbor.  we  obtain  for  our 

minimum 

yn(ceil(d/l))  +  (cei l(d/ l)-1 ) (cei  I  (yn/p)p-yn)  ♦  k 

The  first  term  accounts  for  the  correct  number  of  looo 

traversals  required  to  increase  the  internal  delay  estimate 
beyond  the  external  delay  estimate.  The  last  term  accounts 
for  the  processing  of  the  one  reouired  update  from  the 
external  node.  The  middle  term  accounts  for  the  fact  that 
routing  updates  may  happen  no  more  often  than  one  every  p 
msec  over  each  channel.  Table  2.1  shows  the  removal  time 

for  loops  of  size  up  to  ten  nodes/  using  a  value  of  25  for  x 
and  15  for  k  (see  Chapter  3)  and  assuming  that  the  loop  is 
minimum  delay  estimate  difference  so  that 
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caused  by  a 


cei I (d/l) 


1  . 


Loop 

p  =  6A0 

msec 

p=128 

msec 

Size 

Maxi  mum 

min imum 

'Maximum 

y  i  n  i  m  u  m 

(nodes ) 

(msec) 

(msec) 

2  0  A  Q 

95 

5  04 

95 

< 

2720 

135 

672 

135 

U 

3400 

175 

840 

175 

5 

9  J80 

215 

10  08 

215 

6 

4  7  60 

255 

1176 

255 

7 

5440 

295 

1  344 

295 

r:' 

6120 

335 

1512 

335 

9 

6PUD 

375 

1680 

375 

1C 

7480 

415 

18  48 

415 

Table  2.^.  ARPANET  loop  persistence  time 


In  general  routing  updates  among  neighbors  are  neither 
synchronized  nor  completely  unsynchronized.  Thus  the  ex- 
oecteci  loop  persistence  time  lies  between  the  two  extremes. 
It  is  now  clear  that  loops  do  not  disappear  ouickly/  on  the 
average.  Therefore  it  would  be  useful  to  examine  methcds  of 
prevention  or  at  least  rapid  detection  and  removal. 
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2. A  A  dirgci  agEroach  to  loo£  contLEi 

e®low  we  describe  two  loop  control  algorithms  for  sin¬ 
gle  entry  routing  tables  (as  in  the  ARPANET).  The  first  is 
ping-pong  or  local  loop-free  and  the  second  is  general 
loop-free  . 


2.A.1  A  locai  i22E"flg§  CEUtiDS  (LUE.E) 

The  basic  idea  for  eliminating  local  loops  is  to  ignore 
one's  "old"  information.  Each  node  ignores  routing  informa¬ 
tion  which  points  in  that  node's  direction.  Routing  message 


entries  are  marked  so  that  the  receiving  node  knows  which 
entries  point  toward  it  and  which  do  not.  Only  those  en¬ 

tries  which  point  away  are  used  in  the  update.  More  pre¬ 
cisely  »’  we  add  to  each  entry  in  the  routing  update  message 
the  best  delay  channel  number  c(j#d)#  and  a  neighbor  table  n 
is  appended  to  the  routing  update.  An  entry  n(j#k)  oives 
the  address  of  the  neighbor  connected  to  j  on  channel  k. 
(Fntries  could  be  marked  with  a  single  bit/  but  this  re¬ 
quires  the  sending  of  a  different  routing  message  to  each 
neighbor.)  Each  node  i  performs  the  following  aloorit^'m: 
LL£& 

1  m  =  k  such  that  n(j/k)  =  i 

2  for  each  d 

3  if  c<j/d)  t  m  then  do 

4  t'(i/d)  =  t<j/d)  +  q(j/d)  ♦  h 

5  if  t*(i/d)  <  t(i/d)  then  c(i/d)  =  c'(i/j) 

6  if  c(i/d)  =  c*(i/j)  then  t(i/d)  =  t*(i/d) 

7  end 

In  a  later  section  we  prove  that  this  algorithm  can 
create  no  local  loops. 

2.4.2  A  l22B-l£££  L22SiQ2  ikaSCittlB! 

The  loop-free  routing  algorithm  consists  of  LLFP  at 
one-  and  two-connected  nodes  with  the  addition  of  the  fol- 
lowino  for  three-  (or  more)  connected  nodes  (where  a  node's 
connectedness  is  defined  to  be  the  number  of  directly 
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connected  neighbor  nodes).  When  a  routing  update  indicates 
that  a  change  should  occur  at  a  three-  (or  more)  connected 
node  a  "Loop-check"  control  packet  (LCP)  is  sentr  over  the 
new  channels  one  for  each  destination  which  would  be  effect¬ 
ed  by  this  route  change.  The  idea  being  that  if  the  LCP 
returns  to  the  sender#  then  a  loop  would  be  created  by  such 
a  routing  change.  If  an  end-to-end  acknowledgement  (PCP) 
for  the  Loop-check  packet  returns  to  the  sender#  then  he 
upoates  his  table  and  uses  the  new  route.  Otherwise  (the 
loop-check  packet#  an  " I  * m-check i ng”  packet  (ICP)#  or  noth- 
ina  returns)  he  ignores  the  new  route  until  (possibly) 
another  routing  update  is  received.  Loop-check  packets  are 
routed  through  the  network  in  the  usual  way  (i.e.#  along 
existing  allowed  paths)  except  that  at  any  node  which  is 
also  checking  a  new  route  for  that  same  destination)  such  a 
node  must  send  an  I'm-checking  packet  to  the  source  of  the 
loop-check  packet  and  discard  the  loop-check  packet.  Rout¬ 
ing  message  entries  for  any  destination  site  involved  in 
loop-check  are  specially  marked  so  that  the  information  will 
be  ignored  by  the  receiver  of  the  routing  message.  Stated 
more  precisely  the  assimilation  of  a  routing  message  is  done 
under  the  following  algorithm  for  each  d: 


L£R  csuliQg  mfiisaafi  siaisiiaiifiD 


1 

? 

3 

4 

5 

6 
7 


9 


10 


11 

12: 

13 

14 

15 

16 
17 
IP 

19 

20 
21 
22 

23 

24 

25 


if  x(i^d)  =  NULL  then  do 
if  c(jxd)  =  III  then  do 

if  c<i#d)  =  c*(ixi)  then  do 
local  loop  detected! 
tli^d)  =  t<j/d)  ♦  I 
end 
end 

else  do 

t'fi^d)  =  t(j/d)  +  I 

if  c'(i#j)  =  x(i#d)  then  y(i/d)  =  t*(i/d) 
else  do 

if  t'(ird)  <  y(i/^d)  then  do 
y<i/d)  =  0 
x(i/d)  =  NULL 
end 

if  t'Ci^d)  <  t(i^d)  and  x(i^d)  =  NULL  then  do 
if  i  is  >  2-connected  then  do 
x(i/d)  •=  c'(i^j) 
yCi^d)  =  t’li-'d) 

send  LCP  to  d  via  c*(i/^j)  with  new  seq  no 
end 

else  c(i/d)  =  c'(i^j) 
end 
end 

if  c(i/d)  =  c’fi^j)  then  t(i/d)  =  t'(i/d) 
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end 


where  xCi^d)  is  tne  test  channel  for  destination  d  at  node 
\r  y(i»d)  is  the  test  channel  delay  estimate  to  node  d  from 
node  i/  m  is  defined  as  in  LLFR#  and  t  is  the  local  delay 
ie.q.f  I  =  q(i^j)  +  h).  Some  additional  processing  of  a 
packet  is  required  as  it  arrives  from  a  channel: 

LFR  addillSQSi  jacket  Qrgc^i^iQa 

101  if  ICP  for  i  then  do 

102  if  current  sequence  number  then  do 

103  X (i /d)  =  NULL 

104  y(i^d)  =  C 

105  end 

IC'6  discard  packet 

107  end 

108  if  LCP  then  do 

109  if  source  =  i  then  do 

110  if  current  sequence  number  then  do 

111  xliyd)  =  NULL 

112  y(i/d)  =  0 

113  end 

114  discard  packet 

115  end 

116  if  destination  =  i  then  do 

117  send  RCP  to  source 

118  discard  packet 

119  end 
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If  x(wd)  #  NULL  then  do 


120 

121  discard  packet 

122  send  ICP  to  source 

123  end 

124  end 

125  if  RCP  for  i  then  do 

126  if  current  sequence  number  then  do 

127  t(ixd)  =  y(i/d) 

128  c<i^d)  =  x(i^d) 

129  x<ixd)  =  NULL 

130  y(i/d)  =  C 

131  end 

132  discard  packet 

133  end 

134  if  packet  not  discarded  then  do  normal  packet  processing 

It  is  useful  to  consider  how  these  algorithms  would 
affect  the  ARPANET.  These  aloorithms  appear  to  fit  easily 
within  the  current  ARPANET  system  with  respect  to  the  fol~ 
lowing: 

1.  The  same  routing  message  may  be  sent  to  each  of  a  node's 
neighbors . 

2.  Routing  updates  fit  within  the  current  packet  size  res- 
t  ri ctions . 

3.  The  processing  complexity  is  not  significantly  increased. 

(In  the  case  of  LLFR/  processing  is  certainly  less  com¬ 
plex  than  HDR  or  KHDR.) 
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we  expect  that  the  frequency  of  sending  of  the  I 'm-checki ng 
packet  is  quite  small.  The  frequency  of  both  of  these  new 
types  of  control  messages  may  only  be  tested  by  careful 
measurements  however.  Notice  that  to  avoid  race  conditions/ 
one  must  not  send  a  routing  update  while  in  the  process  of 
updating  his  own  tables  and  routing  messages  must  not  be 
allowed  to  cross  paths  (i.e.s  sent  from  A  to  B  and  froir  P  to 
A  in  the  same  time  span).  This  is  easily  dealt  with  by 
using  a  pair  of  sequence  numbers  in  the  routing  packets. 
Node  A  ignores  B’s  update  unless  it  contains  A*s  last  se¬ 
quence  number  . 

we  expect  that  the  performance  of  LLFP  is  better  than 
HORs  since  it  is  less  complex  yet  guarantees  that  no  local 
loops  occur  (as  we  shall  prove).  For  LFRs  the  overhead  is 
much  higher.  There  is  some  extra  computation  recui red  when 
a  packet  arrives.  This  added  computation  is  insignificant 
when  compared  to  the  slowness  of  routing  chances  and  the 
requirement  that  network  bandwidth  be  used  by  the  loop-check 
packets. 

Routing  chances  at  three-  or  more  connected  nodes  can¬ 
not  occurs  on  the  averages  more  quickly  than  about  12  msec 
per  hop  in  the  new  route  (200  bits  in  a  loop-check  packet 
and  200  bits  in  an  RCP  gives  8  msec  transmission  delay.  An 
additional  2  msec  is  required  for  processing  of  the  packets. 
Add  to  tnat  about  2  msec  in  propagation  delay  per  hop.)  The 
average  network  distance  in  May  was  6.24  hops/  with  a 


■  axiaiuM  of  about  11.  Hence#  one  can  estimate  that  on  the 
averaoe  a  minimum  of  75  msec  Is  required  to  make  a  routing 
change  after  it  has  been  suggested  by  one’s  neighbor.  If 
one  assumes  that  loop-check  packets  are  high  priority  (I.e.# 
next  to  go  out  on  the  channel)  and  that  a  full  packet  has 
just  started  out  on  the  line  when  the  loop-check  packet 
arrives#  then  the  delay  Increases  to  about  60  msec  per  hop. 
This  suggests  that  the  maximum  switch  time  Is  on  the  order 
of  700  msec.  Our  experience  shows  that#  on  the  average#  a 
small  number  (<5)  of  destinations  are  Involved  when  a  switch 
occurs.  At  no  time  would  a  three-connected  node  loop-check 
more  than  about  1/3  of  the  nodes  (less  than  20  nodes  at 
present)  In  the  network  after  a  routing  update.  So  that 
between  50  and  220  msec  of  channel  time  and  5  to  20  buffers 
along  the  way  might  be  utilized. 

An  alternate  method  of  sending  the  loop-check  packets 
could  be  used  to  trade  computation  for  storage  and  line 
utilization.  One  may  send  a  mu  It  1 -dest Inat 1  on  packet  which# 
along  with  the  usual  packet  header#  contains  some  (4 
currently  in  the  ARPANET)  words  of  addressing  information. 
These  addressing  words  consist  of  one  bit  for  each  node  In 
the  network.  The  packet  is  marked  by  the  sender  with  e  one 
bit  for  each  destination  for  which  a  loop-check  Is  required. 
Then  it  is  sent  over  the  new  channel.  Each  node  which  en¬ 
counters  the  packet  is  required  to: 

1.  Send  an  RCP  if  Its  own  bit  Is  on? 

2.  Turn  its  own  bit  off# 
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3.  Send  an  ICP  to  the  source  marking  those  destinations 
being  checked  by  both  this  node  and  the  source; 

4.  Turn  off  the  bits  for  those  destinations  being 


checked  by  both  this  node  and  the  source; 

5.  Discard  the  packet  if  all  addressing  bits  are  now 
off; 

6.  (Possibly)  duplicate  the  packet; 

7.  Mark  the  copies  with  those  destinations  pointed  to 
by  each  of  the  channels  as  indicated  in  his  routing 
table;  and 

S.  Forward  the  packet(s)  on  their  way. 

This  method  would  cut  some  of  the  channel  and  storage  over¬ 
head  at  the  expense  of  some  processing  overhead. 

Pelow  we  present  a  proof  that  the  algorithms  do  prevent 
loops.  In  a  later  section  we  oresent  our  experience  with 
these  algorithms  through  simulation. 

2.5  ^rgalS  al  i.22C~t£S£DSSS 

There  are  a  number  of  definitions  which  facilitate  the 
proving  of  the  loop  prevention  properties  of  these  algo¬ 
rithms.  These  definitions  help  in  the  understanding  of 
these  properties  as  well.  The  algorithm  may  be  described  bv 
a  graph  structure  in  which  there  are  directed  and  non- 
directed  arcs.  The  non-directed  arcs  are  the  lines  between 
neighboring  nodes.  The  directed  arcs  represent  the  direc¬ 
tion  hat  a  packet  to  a  particular  destination  would  be 
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sent.  There  exist  a  set  of  operators  which:  (1)  replace 
directed  arcs  (routing  changes);  (2)  remove  or  create  non- 
oirected  arcs  (network  failures  and  recoveries). 

We  define  a  network  to  be  a  graph  G  consisting  of  a 
set  X  of  nodes  related  by  three  relations: 

n(eighbor)»  r(oute)/  and  (te)s(t  route) 

If  x1/  is  a  neiohbor  of  x2  in  the  graph  G  then  x1  is 
inthesetn(x2)  and  x2  is  in  n(x1).  A  node  x  is 

k-connected  if  and  only  if  there  are  exactly  k  distinct 
nodes  in  n(x).  Ue  will  refer  to  a  k-connected  node  as 
mu  1 1 i -con nec t e d  if  k>2 .  The  next  node  in  the  route  to  a 

particular  destination  xd  from  x1  is  r(x1)  .  We  shsll 
fix  the  destination  xd  here#  and  drop  reference  to  it  for 
simplicity.  All  succeeding  definitions  (and  proofs)  refer 
implicitly  to  a  particular  destination.  Loop-check  packets 
from  node  x  are  routed  to  the  test  route  s(x)  'rom  x 
Note  that  s(x)  =  NULL  for  all  nodes  except  those  in  loop- 

check  mode.  A  loop/  of  order  k  /  is  defined  to  be  the  fol¬ 
lowing  condition: 

2.1: 

At  some  instant/  there  exists  a  set  Y  =  (y1 / v2/ . . . /yk ) / 
k>1/  a  subset  of  X/  such  that  xd/  the  destination/  is  not 
in  Y  and 


r(yi) 

=  y(i+1) 

for  i  = 

1/.../  k  -  1  and 

r(yk) 

=  yi  . 

Notice  that 

this  is 

static 

(for  possibly  a  short  time) 
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condition  among  the  routing  tables  of  several  nodes.  We  do 
not  intend  to  include  in  this  definition  transient  condi¬ 
tions  under  which  packets  may  actually  loop  Ce  .q .  r  line  or 
node  fai lures)  . 

A  routing  transformation  T  on  G  changes  the  route  or 
test  route  of  a  single  node  pair-  i.e./’ 

1(G)  is  defined  as  follows: 

a)  Tlr(x))  =  r(x)  for  x  not  =  xOr 

T{r(xO))  =  r*(xO)  for  some  node  xOr  and 
T(s<x))  =  s(x)  for  all  nodes  x;  or 

b)  T(r(x))  =  r(x)  for  all  x  not  =  xO/ 

T<r(xO))  =  slxO)  for  some  node  xO/ 

T(s(x))  =  s(x)  for  X  not  =  x0»  and 
T<s<xO>)  =  null;  or 

c)  T<r(x))  =  r(x)  for  all  x  not  =  xO/^ 

T(s(x))  =  s(x)  for  X  not  =  xC/  and 

T(s(xO))  =  NULL  for  some  node  xH;  or 

o)  T(r(x))  =  r(x)  for  all  nodes  xr 

T(s(x))  =  s(x)  for  X  not  =  xH,  and 
T(s(xO))  =  s'(xO)  for  some  node  xO. 

In  case  (a)  the  route  from  node  xO  is  changed  to  a  neighbor 
which  is  not  the  test  route  (which  can  happen  only  at  one- 
or  two-connected  nodes).  In  case  (b)  a  new  route  from  node 
xO  is  chosen  to  be  the  test  route  following  a  successful 
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test.  In  case  (c)  the  test  route  from  node  xO  is  dropped 
following  an  unsuccessful  test.  In  case  (d)  a  test  route 
from  node  xO  is  established.  (Routing  transformations  which 
would  occur  simultaneously  in  an  operating  environment  will 
be  ordered  by  node  number^  say#  and  be  executed  sequentially 
here.)  We  claim  that  LLFR  and  LFR  have  T  with  the  following 
property . 
p£2B££l^  2.1: 

c.  If  r(x)  =  y/  then  T(r(y))  not  =  x; 

b.  if  r(x)  =  yr  then  T(s{y))  not  =  x; 

c.  if  six)  =  yt  then  T(r(y))  not  =  x/  and 

d.  if  s(x)  =  yr  then  T(s(y))  not  =  x. 

Proof  of  Property  2.1: 

Ltffi 

Property  2.1  (a)  is  guaranteed  by  step  3  in  the  LLFR  aloo- 
rithm.  Property  2.1  (b)^  (c)  and  (d)  are  vacuously  satis¬ 
fied  by  LLFR  since  no  test  routes  are  ever  established. 

LFg  For  nodes  y  which  are  two-  (or  less)  connected  no 
test  routes  are  ever  established/^  thus  proving  parts  b  and 

d.  Part  a  is  guaranteed  by  steo  ^  in  the  LFR  aloorithm. 
Part  c  is  similarly  guaranteed  by  step  1.  For  nodes  y  which 
are  three-  (or  more)  connected  parts  b  and  d  are  guaranteed 
by  steps  £  and  1  respectively,  lu It i -connected  nodes  create 
actual  routes  from  successful  test  routes.  huring  the  test¬ 
ing  of  8  route  by  a  node/  his  neiqhoor  (along  that  route)  is 
restricted  from  establishino  a  new  route  opposite  to  the 
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test  route  by  step  1.  Hence  parts  a  and  c  are  true  for 
multi-connected  nodes  as  well. 

G .E. D. 

LfiffilS  ^*1:  Loops  of  order  2  cannot  occur  as  a  result  of  the 
application  of  LLFft  or  LFk. 

Proof:  This  is  a  trivial  consequence  of  Property  2.1.  Sup¬ 

pose  that  there  exists  a  loop  of  order  2  after  the  applica¬ 
tion  cf  some  T  with  Property  2.1.  Then  there  exist  neigh¬ 
bors  X  ana  y  such  that: 

T(r(x))  =  y  and  T(r(y))  =  x  (*) 

In  this  case/  there  are  4  possible  states  prior  to  the  ap¬ 
plication  of  T: 


1) 

r  <  X ) 

=  y 

and 

r  ly )  =  xe 

2) 

r(x) 

=  y 

and 

rCy)  not  =  xr 

3) 

r(x) 

not 

=  y 

ana  r(y)  =  xr  and 

4) 

r  (  X ) 

not 

=  y 

and  r<y)  not  =  x. 

State  4  cannot  exist  since  T  may  modify  only  one  r  (this  is 
where  the  synchronizing  of  neighbors  is  important  in  an 
operating  network).  State  1  is  a  oreviously  existing  loop 
and  hence  was  not  caused  by  T.  States  2  and  3  are  sym¬ 
metric.  Thus  we  consider  only  state  2.  Since  T  has  Proper¬ 
ty  2.1^  and  r(x)  =  y  /  then  T(r(y>)  not  =  x  /■  in  contrad¬ 
iction  to  the  supposition  at  the  beginning  of  the  proof. 
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Thus  (*)  cannot  occur. 


C .E. D. 


we  have  proved  the  follouino  theoreti. 

I!1SQ££0  The  LLFR  alqorithm  is  free  of  local  looos. 

We  now  concentrate  on  LFR. 

k§693  d'i-  A  one- c onnec t e d  node  cannot  participate  in  a 
multi-node  loop  (i.e./  a  loop  of  order  k>?). 

Proof:  A  one-connected  node  has  only  one  neighbor.  If  a 

one-connected  node  participates  in  a  loop  it  must  be  of  ord¬ 
er  2 . 

0 .E. D. 


USSISIS  Z’i'  Loops  of  order  greater  than  2  cannot  be  caused  by 
a  routing  change  at  a  two -connect e d  node. 

Proof:  Suppose  a  loop  of  order  k  >  2  is  created  by  a 
transformation  at  a  two-connected  node  y  .  The  loop  then 
satisfies  Condition  2.1  in  the  following  way: 


T(r(x1)) 


r(x1) 


=  xii 


T(r(xi))  -  r(xi)  =  y 
T(r{y))  =  r*(y)  =  x<i  ♦  1) 


T(r(x(k 

-  2)) 

=  r<x(k  -  2)) 

=  x(k 

-  1) 

T  (  r  (  X  (  k 

-  in 

=  r(x(k  -  1)) 

=  x1 

Also 

we  have 

that 

r(y)  not  =  x(i 

+  1  )  . 

In 

f  act  ^ 

since  y 

1  s 

two-conne  cted 

we  have  r(y)  = 

x  i  . 

Put 

r  ( X  1 )  = 

y  .  This 

is  a 

ore-ex i st ing 

loop  of  order 

2  which 

by 

Lemma  ? 

.1  cannot 

occur . 

C.E.D. 


kS[S&3  ^.4:  Loops  of  order  k  >  ?  cannot  be  brolren  by  a  two- 
connected  node  . 

Proof:  Suppose  a  loop  of  order  k  >  2  contains  a  node  of 
order  2.  Any  routing  change  at  that  node  alone  would  be  a 
violation  of  Property  2.1. 

0. E. D. 
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LSSfS  2.^:  If  a  routing  transforaat ion  causes  a  loop  to 
occur  then  the  node  xO  #  for  which  T(r(xO))  not  =  r(xO)# 
is  contained  withi*^  the  loop. 

Proof:  Suppose  after  the  application  of  T  there  exists  a 
loop  aiiono  a  set  Y  of  nodes.  If  xO  is  not  in  'i  r  then 
for  all  y  in  Y  we  have  T<r(y))  =  r(y)  /  whirh  implies 
that  the  loop  existed  prior  to  the  application  of  T.  Hencex 
xu  must  be  in  Y  . 

O.P.D. 

Lemma  2.5  shows  that  the  loop-check  packet  has  the  pos¬ 
sibility  of  being  returned  to  its  sender/  if  a  loop  would 
have  been  caused  by  that  routing  change. 

IbSSISQ!  Z’i'  Loops  cannot  occur  as  a  result  of  LFP. 

Proof:  U'e  have  previously  proved  that  2-order  Itops  do  not 
occur  (Lemma  2.1)  and  that  k-order  loops  for  k  >  2/  are 
independent  of  (i.e./  cannot  be  caused  or  removed  by)  two- 
connected  (or  one-connected)  nodes  (Lemmas  2.2/  2.2  and 
2.4).  We  shall  now  show  that  loops  of  order  k  >  2  cannot  be 
caused  by  transformations  at  mult  i -connec  t  ed  nodes.  Suppos'* 
that  we  restrict  the  number  of  concurrent  mul ti -connected 
node  transformations  to  one.  That  is  to  say/  only  one  node 


may  test  any  particular  destination.  Suppose  that  a  loop 
would  be  created  by  a  transformation  at  node  x*  a  multi- 
connected  node.  Since  x  is  mu  1 1 i - conne c t ed  it  is  reouired 
to  test  the  new  route  with  a  loop-check  packet.  There  are 
four  possible  outcomes  from  the  sendinq  of  this  loop-check 
packet: 

1.  The  loop-check  packet  returns# 

2.  An  I'm-checkinq  packet  returns  (If  some  other  node 
is  checking)# 

3.  Nothing  returns  (If  the  message  is  lost  or  becomes 
trapped  in  a  loop  independent  of  x#  should  one 
occur)#  or 

4.  An  RCP  returns. 

In  cases  1  through  3  no  switch  will  be  made  in  the  route 
+rom  X#  and  hence  no  loop  created.  In  case  4  we  have  the 
fact  that  the  loop-check  packet  from  x  reached  its  destina¬ 
tion.  Since  no  other  routing  changes  occur#  any  of  x*s 
packets  will  also  reach  the  destination  by  following  the 
exact  route  that  the  LCP  took.  Therefore  no  loop  exists. 
Hence  a  routing  transformation  at  only  one  mult i -connected 
noae  cannot  cause  a  loop. 

we  now  proceed  by  induction  on  the  number  of  concurrent 
mul t i -connect ed  node  routing  switches.  Assume  that  n-1  or 
less  concurrent  switches  cannot  cause  a  loop.  Suppose  n 
concurrent  switches  ha^e  caused  a  loop.  Consider  the  ac¬ 
tivity  of  node  x#  that  node  which  is  last  to  start  loop 
check.  Again  the-re  are  four  possible  responses  te  x's  LCP. 


They  are  as  listed  above.  If  2  or  3  occurs  then  x  will 
not  switch  routes  which  contradicts  the  supposition  that  n 
concurrent  switches  caused  the  loop.  Therefore  we  need  only 
consider  response  4.  There  are  two  possible  ways  in  which 
this  may  occur. 

(a)  x's  LCP  arrives  at  none  of  the  other  n-1  nodes 
involved#  or 

(b)  x's  LCP  arrives  at  (at  least)  one  of  the  other  n-1 


nodes  after  that  node  has  completed  switc^'ing. 


In 

case  (a) 

the  switch 

at  X 

must 

be  independent  of 

the 

loop. 

No 

switch 

occurs  at 

any 

of 

the  nodes 

wh  i  ch 

X  *  E 

LCP 

t  reverses . 

Therefore 

X 

c  annot 

be  part  of 

the  loop# 

since 

all 

packets 

from  X  will 

reach  the 

destination  as 

did 

the 

LCP.  Therefore  the  switch  at  x  could  not  have  been  required 
to  form  the  loop.  That  is  the  loop  would  have  been  created 
whether  or  not  x  switches.  This  contradicts  the  supposition 
again.  In  case  (b)  we  have  that  the  loop  could  have  been 
caused  by  less  than  n  concurrent  switches#  which  contradicts 
the  induction  hypothesis.  Hence  there  exists  no  n  such  that 
n  concurrent  transformations  at  mu  It i “connected  nodes  causes 
a  loop. 

Q.E.D. 

An  adaptive  routing  procedure  should  not  be  vulnerable 
to  \network  component  failures  (i.e.#  line  or  node  outages). 
We  have  not  shown  that  the  algorithm  is  invulnerable  to  net¬ 
work  failures.  It  is  clear#  however#  that  a  loop  cannot  be 
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caused  by  a  failure  (i.e.^  the  renoval  of  an  arc).  A 
recovery  cannot  cause  a  loop  either  because  when  a  line 
comes  up  it  has  no  direction  in  a  path.  Therefore^  no  loops 
are  created  by  a  failure  or  a  repair.  However^  on  ■failure 
(or  repair)  some  routes  must  change.  This  may  cause  flood¬ 
ing  of  the  network  with  LCP’s  which  is  clearly  one  of  the 
undesirable  features  of  LFR. 

2.6  Simyiatign  re^ul^tsi 

In  order  to  examine  the  performance  of  these  routing 
procedures  we  here  present  the  results  of  simulation.  The 
simulation  was  performed  under  the  following  conditions: 

Topology:  ARPANET  (June  1975)  modified  to  excluae 

satelli te  links  . 

Lines:  ARPANET  capacities  (mostly  50  kh/s/  some  230.4 

kb/s) . 

Nodes:  Infinite  storage^  two  processor  speeds 

(316  -  1.6  usec/cycle  and  516  -  .96  usec/cycle). 

Traffic  pattern:  Uniform  traffic  matrix  (i.e.#  same 
intensity  between  all  pairs  of  nodes).  Arrivals 
to  the  network  form  a  Poisson  orocess.  Exactly 
the  same  traffic  was  generated  for  each  of  the 
several  routing  schemes  examined. 

Piessage  length  distribution:  Exponential  truncated  at 
3C64  bits^  with  a  mean  of  122  bits.  (The  mean  is 
one  half  of  that  reported  in  CKlei  74D  to  allow 
for  RFNM's  which  have  zero  text  bits.  Py  reducing 
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the  average  message  size  ve  attempt  to  more  close¬ 
ly  model  the  actual  packet  size  distribution  which 
consists  primarily  of  data  packets  and  RFNW's  in 
the  ARPANET.) 

The  results  of  simulation  are  shown  in  Figure  2.3. 
Network  wide  mean  (one  way)  delay  is  plotted  as  a  function 
of  relative  offered  traffic  intensity  (load)  in  this  figure. 
We  observe  that  all  algorithms  produce  the  same  network  wide 
mean  delay  up  to  an  offered  traffic  load  of  5  (5C0  packets 
per  second).  At  10  we  see  that  LPR  blows  up  (i.e.»  averaoe 
delay  >  5G&  msec).  (Actually  the  simulation  failed  to  reach 
equilibrium  at  this  load.)  This  is  due  to  the  creation  of 
many  loops.  Also  at  load  10  HOR  and  LLFR  perform  better 
than  LFR.  (At  this  load  no  multinode  loops  are  yet  prevent¬ 
ed).  The  most  interesting  result  is  that  while  both  HDR  and 
LFR  blow  up  at  a  load  of  14/  LLFP  continues  with  finite 
delay  to  at  least  16  (the  highest  load  simulated).  LLFR 
exceeds  the  performance  both  in  terms  of  delay  and 
throughput  at  all  levels  of  offered  load.  This  is  not  too 
surprising  since  multinode  loops  occur  very  infrequently. 
Therefore  the  overhead  which  LFR  requires  to  guarantee 
loop-freeness  is  ill  spent  in  the  topology  and  traffic  pat¬ 
tern  of  the  simulation.  Since  local  loops  may  (do)  occur 
with  HPR/  it  performs  worse  than  LLFR  which  prevents  such 


Mean  delay  (msec) 


Offered  traffic  intensity  (100  messages/sec) 


Figure  2.3.  Comparative  performance  of  adaptive  routing  procedures  (simulation). 
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There  Mere  no  loops  observed  below  a  load  of  10  in  this 
simulation.  In  different  topologies  (and  non-unifortr  traff¬ 
ic  rjies)  it  is  possible  to  create  loops  at  lower  loads.  fit 
load  10^  301!  of  the  routing  changes  at  two-connected  nodes 
resulted  in  local  loops  for  LPR.  HDR  had  no  loops  ur  to  a 
load  of  14  at  which  point  21%  of  the  t wo -conne ct ed  node 
routing  changes  resulted  in  local  loops.  The  number  of 
loop-traps  was  unfortunately  not  measured  in  the  simulation. 

As  for  the  frequency  of  packet  looping^  there  was  none 
until  load  10  for  LPR^  load  14  for  HDR#  and  load  15  for 
llFR.  As  expected#  no  looping  other  than  LCP's  was  observed 
for  the  LFR  simulation.  The  percent  of  packets  which  looped 
was  4.8#  2.6#  and  .8  respectively.  (The  values  for  LPP  and 
HDR  are  subject  to  question  since  the  simulation  did  not 
reach  equilibrium  in  these  cases.) 

Both  LLFR  and  LFR  prevented  local  loops  beoining  at  a 
load  of  10.  LFR  began  to  prevent  multinode  loops  at  a  load 
of  12.  The  average  delay  to  receive  an  RCP  increased  with 
Load  (as  expected).  The  average  had  a  range  of  from  246  to 
275  msec.  T^e  average  number  of  LCP’s  per  test  (i.e.#  the 
average  number  of  destinations  involved)  remained  fairly 
small  but  increased  with  load  from  a  minimum  of  1.1  to  a 
maximum  of  2.8.  This  suggests  that  mu  1 1 i add  re ssed  LCP's 
would  not  help  efficiency  significantly.  It  also  suggests 
that  LFR  has  the  nasty  habit  of  increasing  its  overhead  when 
the  network  is  heavily  loaded. 

In  the  simulations  we  have  assumed  that  each  of  the 
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routing  update  schemes  requires  the  same  amount  of  process¬ 
ing  as  does  HDR.  This  is  clearly  not  the  case  indicating 
that  at  least  for  LLFR  we  have  shown  a  lower  bound  on  per¬ 
formance  with  respect  to  HDR.  It  should  he  noted  that  LLFR 
and  LFR  eliminate  the  need  for  performing  any  computation 
(beyond  the  initial  test)  for  a  fraction  1  -  1/lc  of  the 
entries  in  the  routing  table  on  the  average^  where  k  is  the 
connectivity  of  the  neighbor.  The  issue  of  processing  over¬ 
head  is  discussed  more  fully  in  the  next  chapter. 

2.7  CgnciusiQns 

Our  intuition/  which  is  now  supoorted  by  simulation/ 
tells  us  that  multi-node  loops  occur  very  infrequently  in  a 
"reasonable”  topology  ond  traffic  pattern.  Therefore/  since 
LFR  is  so  complicated/  its  practical  worth  is  questionable. 
However/  it  is  interestino  to  know  that  such  a  scheme  ex¬ 
ists. 

Local  loops/  on  the  other  hand/  occur  much  more  fre¬ 
quently  and  in  fact  were  observed  in  the  ARPANET  as  well  as 
in  simulation.  Therefore  LLFR  is  of  practical  significance. 
Indeed  the  simulation  results  indicate  that  LLFR  is  the  best 
among  the  procedures  studied  so  far.  In  Chapters  3  and  4 
other  routing  procedures  are  investigated.  We  defer  our 
recommendations  regarding  routing  until  after  Charter  A. 
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CHAPTER  6 


SYSTEM  PRIORITIES 


3-1  IniCQsJgfiiiaQ 

The  inclusion  oT  levels  of  priority  within  a  system 
allows  that  system  to  give  different  levels  of  service  to 
different  classes  of  customers.  This  is  a  useful  property 
of  a  system  which  is  to  support  a  variety  of  activities 
simultaneously.  The  ARPANET#  for  example#  was  origimally 
designed  to  support  both  interactive  (term inal-to- computer 
ano  computer-to-terminal )  traffic  as  well  as  moving  large 
amounts  of  data  from  place  to  place  in  the  network 
( comouter-tO“Computer  communication).  BBN  has  thoughtfully 
provided  a  priority  structure  within  the  network  nodes.  In 
this  chapter  we  partially  examine  that  priority  structure 
and  its  effect  or  performance.  The  result  of  this  examina¬ 
tion  points  out  the  general  rule  that  careful  consideration 
must  be  given  to  priority  at  al^  points  of  service  within 
the  system  in  order  to  guarantee  the  desired  level  of  ser¬ 
vice  to  each  customer  class. 

Initial  analysis  and  design  of  packet  switched  networks 
assumed  that  the  nodal  processing  time  was  small  and  fixed 
CKlei  64]#  CFran  70]#  CHear  70]#  CKlei  70]#  CFran  723  and 
others.  often#  in  fact#  this  contribution  to  delay  has  been 
neglected  in  comparison  to  the  time  spent  waiting  for  and 
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using  the  channels  In  the  network.  In  short»  the  transmis¬ 
sion  lines  were  the  bottleneck  in  the  system.  The  ARPANET 
in  fact  was  designed  with  this  underlyina  assumption.  (f^ne 
must  in  general  attempt  to  fully  utilize  the  most  expensive 
component  in  a  system.)  Here  we  present  measurements  which 
demonstrate  that  nodal  processing  time  is  no  longer  small  or 
fixed  in  the  current  ARPANET.  Nodal  processing  delay  is  in 
fact  a  significant  portion  of  the  overall  network  delay. 
This  need  not  be  the  case.  The  problem  is  one  of  assigning 
the  proper  priority  to  tasks  within  the  processor.  This  we 
show  through  the  use  of  analysis  and  simulation. 

3.2  tisasuced  Rgsyl^S 

A  large  increase  in  network  wide  mean  round-trip  delay 
was  noticed  between  the  "weeklong"  data  collections  of  Au¬ 
gust  1973  reported  in  CKlei  and  May  1974.  While  the 
traffic  characteristics  did  not  change  significantly/-  delay 
increased  from  93  to  249  msec.  This  delay  (while  fairlv 
small)  exceeds  the  ARPANET  specification  ot  ?00  msec  and  has 
remained  at  about  this  level  even  as  late  as  March  1977  when 
another  collection  revealed  a  mean  delay  of  22 E  msec.  Table 
3.1  summarizes  the  information  derived  from  the  three  week- 
long  collections.  There  are  three  factors  (listed  in  the 
table)  which  would  naturally  have  caused  an  increase  in 
delay  between  August  1973  and  May  1974.  Notice  that  in  Mav 
1974  (a)  messages  contained  slightly  more  packets  on  the 
average/  (b)  messages  traveled  longer  distances  (in  hops)  on 


the  average  and  (c)  the  mean  channel  utilization  (with  over¬ 
head)  was  larger  than  in  August  1973.  The  reason  for  the 
sharp  increase  in  channel  utilization  is  due  to  the  fact 
that  the  frequency  of  routing  updates  was  increased  approxi¬ 
mately  by  a  factor  of  five.  All  of  these  three  factors  are 
accounted  for  in  the  model  of  CKlei  74D.  Yet  the  model  is 
unable  to  predict  all  of  the  increase.  while  actual  delay 
increased  by  168%  and  145%#  the  model’s  predictions  in¬ 
creased  by  only  95%  and  44%  from  August  1973  to  f^av  1974  and 
from  August  1973  to  March  1977  respectively.  fhis  is  due 
(at  least  in  part)  to  the  fact  that  the  model  of  CKlei  743 
assumed  a  small  and  fixed  nodal  processing  delay  which  is  no 
longer  true. 


August  1973 

May  1974 

March  1977 

Input  rate  (pkts/sec) 

51 

44 

io» 

•^ean  bits/msg 

243 

234 

266 

Pean  pkts/msg 

1.11 

1.12 

1  .1« 

Mean  traffic  weighted 
shortest  hop  path 

3.24 

4.46 

2.93 

f-ean  channel 
utilization 
with  overhead 

.071 

.2C4 

.237 

Mean  channel 
utilization 
without  overhead 

.(;077 

.011 

.012 

r-.ean  measured 
round-trip  delay  (msec) 

93 

249 

229 

Cklei  743  model 

delay  prediction  (msec) 

73 

142 

101 

Table  3.1  ARPANET  traffic  and  delay  summary 


In  examining  the  results  of  measurement  experiments 
involving  the  use  of  packet  trace  iKlei  7balf  it  becomes 
clear  that  the  processina  delay  has  a  measured  value  which 
is  much  larger  than  the  available  estimates.  Some  examples 
follow.  Table  3.2  shows  the  results  of  an  experiment  in 
which  packet  trace  was  collected  with  a  frequency  of  ?56 
(i-e./  a  trace  block  was  generated  for  every  Z56th  racket 
traversing  a  node)  from  several  nodes  in  the  ARPANET.  The 
data  was  collected  during  two  approximately  ?0  hour  intei — 
vals  in  March  1975.  Statistics  for  individual  output  lines 
are  included  (though  one  would  not  expect  processing  delay 
to  vary  with  output  line).  The  first  six  column  headings 
are  clear/  but  the  last  three  require  some  explanation. 
Smpl-Size  (sample  size)  refers  to  the  actual  number  of  pack¬ 
ets  used  to  arrive  at  the  previously  listed  statistics 
(i.e./  Mean/  SDev/  Min/  Max).  The  mean/  standard  deviation/ 
minimum/  and  maximum  are  expressed  in  milliseconds  (msec). 
Only  sto re- and-f orwa rd  packets  which  did  not  originate  at 
the  From-Node  may  be  included  in  the  Smpl-Size/  as  packets 
from  a  HOST  do  not  have  their  *'time-in"  recorded  properly 
and  thus  are  excludeo  from  the  statistics.  They  are/  howev¬ 
er/  included  in  the  To-Ch-Tot  (to  channel  total)  which  gives 
the  total  number  of  observed  packets  to  be  routed  on  that 
output  channel.  The  To-Task-Tot  (to  Task  total)  entry  is 
the  total  number  of  observed  packets  (  s  to  re-and- Fo  rwa  r  c*  and 
reassembly)  which  were  processed  by  the  "Task"  CMcuu  722 
rout ine  in  the  IMP. 


From 

Node 

To 

Node 

Mean 

SDev 

Mi  n 

Max 

Smpl 

Size 

To-Ch 

Tot 

T  o-T  ask 
Tot 

6 

KIT 

31 

CCAT 

2.97 

4.20 

.4 

36.1 

3349 

3724 

p05P 

6 

MIT 

47 

yPAT 

3.12 

4.46 

.4 

36.4 

3260 

4f"P1 

»958 

6 

MIT 

44 

MIT2 

3.36 

4.68 

.4 

2».3 

230 

4C2 

8958 

5 

BBN 

31 

CCAT 

3.10 

4.51 

.4 

33.9 

28  52 

3119 

8065 

5 

BBN 

50 

RCC 

3.06 

4.21 

.4 

25.0 

1345 

1746 

8065 

5 

BBN 

49 

RCCT 

5.15 

4.39 

.4 

30.4 

1581 

2149 

8  065 

U 

CMU 

3F 

PURO 

3.61 

4.73 

.4 

32.6 

1959 

2329 

5930 

U 

CKU 

27 

BEL  V 

3.13 

4.60 

.4 

33.3 

2071 

2165 

5030 

14 

C.-IU 

1» 

RADT 

3.31 

4.74 

.4 

23. P 

645 

674 

5930 

29 

ABRD 

46 

RUTT 

2.67 

3.78 

.4 

30.6 

1592 

1610 

4720 

29 

APRO 

19 

NBST 

2.67 

3.67 

.4 

23.3 

1691 

1603 

4720 

29 

AbRC 

27 

BELV 

3.02 

4.00 

.4 

23.9 

1X46 

1306 

4720 

11 

STAN 

16 

AMES 

2.14 

3.15 

.  4 

26.0 

2027 

2484 

5  020 

11 

STAN 

22 

ISI 

2.07 

2.96 

.4 

23.4 

1'535 

2180 

5020 

22 

ISI 

11 

STAN 

3.43 

5.39 

.4 

38.1 

572 

1437 

9096 

22 

ISl 

4P 

AFMT 

3.59 

4.60 

.4 

27.3 

544 

1311 

0096 

22 

ISI 

52 

ISIT 

3.46 

4.83 

.4 

26.6 

577 

3683 

9096 

1 

UCLA 

8 

SDC 

2.81 

4.13 

.3 

29.5 

1210 

1486 

4346 

1 

UCLA 

3 

UCSB 

2.90 

3.78 

.4 

20.7 

937 

1PC1 

4346 

1 

UCLA 

35 

UCSO 

2.73 

3.72 

.3 

22.6 

1121 

1508 

4346 

16 

AMES 

15 

AMST 

4.19 

5.81 

.3 

40.6 

983 

1282 

4933 

16 

AMES 

45 

KOFF 

3.71 

4.84 

.4 

26.8 

12P4 

1403 

49*5 

16 

AMES 

36 

HAWT 

3.76 

5.16 

.4 

31.6 

410 

463 

4983 

16 

AMES 

11 

STAN 

4.18 

5.40 

.4 

31  .5 

1427 

1543 

4983 

2 

SRI 

51 

SRI3 

3.02 

4.08 

.4 

25.5 

578 

1^69 

4852 

2 

SF  1 

32 

XRCX 

2.96 

4.15 

.4 

27.6 

1113 

1301 

4852 

2 

SRI 

21 

LLL 

2.89 

4.20 

.4 

24.8 

1041 

1728 

4852 

1C 

LL 

44 

KIT2 

1.80 

2.77 

.4 

14.3 

73 

119 

265 

10 

LL 

18 

RAOT 

2.46 

4.00 

.4 

26.4 

PI 

94 

265 

Table  3.2(a).  Observed  processing  delay  in  msec 

(516  I*IP5) 
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From 

Node 

To 

Node 

Mean 

SDev 

Min 

Max 

Smpl 

Size 

To-Ch 

Tot 

To-Task 

Tot 

34 

LBL 

21 

LLL 

11  .5 

11.3 

.7 

62.2 

34PS 

3402 

10147 

34 

LBL 

4 

UTAT 

ID. 6 

10.8 

.7 

67.7 

4867 

4056 

1  C  1 7 

34 

LBL 

45 

MOFF 

12.2 

11.5 

.7 

64.0 

1477 

1610 

1014  7 

4 

UTAT 

34 

LBL 

17.1 

30.7 

.7 

3  23.0 

4769 

5017 

10534 

4 

UTAT 

12 

ILL 

17.1 

31.2 

.7 

333.2 

4693 

5102 

1C534 

43 

TVMT 

32 

XROX 

18.2 

16.8 

.7 

90.9 

274 

2724 

5200 

43 

TYMT 

33 

FNWT 

18.2 

17.1 

.7 

71.2 

222 

630 

5200 

23 

USCT 

f> 

SDC 

12.2 

13-1 

.7 

76.2 

800 

1647 

4559 

23 

USCT 

25 

DOCT 

12.6 

13.0 

.7 

81.6 

090 

1559 

4559 

25 

DOCT 

23 

USCT 

10.8 

11.7 

.7 

62.9 

1041 

1054 

2096 

25 

OOCT 

24 

GWCT 

1 1  .4 

11.7 

.7 

62.8 

1020 

1038 

2096 

28 

ARPT 

20 

ETAT 

9.70 

11.1 

.6 

54.7 

351 

644 

1114 

2a 

ARPT 

17 

MTRT 

10.6 

10.6 

.  7 

50.6 

287 

308 

1114 

27 

OELV 

14 

CMU 

9.88 

10.0 

.7 

57.8 

1247 

1300 

2791 

27 

BELV 

26 

SOAC 

9-94 

10.8 

.7 

52.3 

391 

391 

2791 

27 

8ELV 

29 

A6RD 

9.18 

9.94 

.7 

51.9 

1061 

1073 

2791 

44 

MIT2 

6 

MIT 

5.47 

6.64 

.7 

32.5 

113 

219 

391 

44 

MIT2 

10 

LL 

4.67 

5.87 

.7 

26.5 

125 

127 

391 

Table  3.2(b).  Observed  processing  delay  in  msec 

(316  IMPS  &  TIPs) 


Only  a  portion  of  the  total  processing  delay  is  meas¬ 
ured  by  the  trace  mechanism^  viz:  (a)  part  of  the  Modem-to- 
IMP  routine  processing  time/^  (b)  the  time  spent  waiting  on 
queue  for  the  Task  routine^  and  (c)  most  of  the  Task  routine 
processing  time.  The  sum  of  (a)/  (b)^  an  (c)  is  given  by 
the  difference  between  T(Z)  and  T(1)  in  the  trace  block.  Pv 
the  estimates  in  CMcQu  721  this  value  would  have  a  minimum 
of  less  than  150+250=400  cycles  (approximately  .3P  msec  for 
a  516  IMP  and  .64  msec  for  a  316  IMP).  The  observed  minimum 
values  confirm  these  estimates. 
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Table  3.2  has  been  separated  into  two  parts  with  516 
IMPS  listed  first  followed  by  the  316  IMPs.  There  is  a 
clear  distinction  between  these  processor  types  (not 
surprising  since  one  runs  at  about  1.67  times  the  speed  of 
the  other).  (Not  as  clear  a  difference  was  observed  between 
those  nodes  with  VDH  CBBN  69]  and/or  TIP  COrns  721  software 
and  those  without.)  In  both  cases  we  see  a  wild  variation  in 
processing  delays  with  mean  values  from  4.5  to  1C. 5  tines 
the  minimum  for  516s  and  from  6.7  to  26  times  the  minimum 
for  316s.  This  phenomenon  is  nearly  independent  of  time  of 
day  as  shown  in  Tables  3.3  and  3.4/  which  show  the  observa¬ 
tions  of  two  selected  nodes  over  seven  time  intervals.  (The 
rather  odd  time  intervals  correspond  to  arbitrary  file  boun¬ 
daries  and  provide  similar  sample  sizes.)  Notice  again  the 
large  variation  in  processing  delay  and  the  very  high  mean 
values.  Notice  also  that  the  mean  values  remain  high  even 
under  the  more  lightly  loaded  evening  hours/  indicating  (hut 
not  proving)  that  this  effect  is  not  due  to  the  competition 
of  packets  for  the  services  of  the  Task  routine. 

For  the  model's  predictions  in  Table  3.1  we  have  as¬ 
sumed  an  average  processing  time  of  less  than  1  msec  per 
hop.  Taking  a  rough  average  of  the  values  in  Table  3.2  we 
find  that  average  processing  delay  for  516  IMPs  is  about  3 
msec  and  about  11  msec  for  316  IMPs.  Roughly  one  third  of 
the  IMPS  in  May  1974  were  516  IMPs.  Assuming  a  uniform 
spread  of  the  traffic/  the  average  nodal  processing  delay  is 


more  than  8  msec-  If  we  add  7  msec  per  hop  to  the  predicted 


delays#  we  would  expect  to  obtain  a  better  prediction. 
Doing  so  we  arrive  at  a  value  of  204  msec  for  May  1974  and 
142  for  ^arch  1977.  For  '«3y  1974  this  new  prediction  is 
slightly  better  than  was  the  original  prediction  for  August 
1973.  The  March  1977  prediction  is  not  as  good.  One  reason 
may  be  the  flow  control  mechanism  which  was  changed  signifi¬ 
cantly  between  May  1974  and  March  1977.  Delay  is  measured 
in  such  a  way  to  include  the  delay  due  to  flow  control# 
while  there  is  no  provision  for  this  in  the  model. 


From 

To 

Mean 

S  Dev 

Min 

Max 

Smp  1 

To-C  h 

To- Task  Time 

Node 

Node 

Size 

Tot 

Tot 

Peg  i  n 

End 

22 

IS  I 

11 

STAN 

4.67 

7.12 

-4 

38.1 

118 

254 

1404 

0952 

1111 

22 

ISl 

48 

AFWT 

4.06 

4.58 

.4 

17.5 

82 

286 

1404 

0952 

1111 

22 

IS! 

52 

ISIT 

3.00 

3.76 

.4 

15.9 

125 

484 

1404 

095? 

1111 

22 

IS! 

11 

STAN 

4.24 

5.87 

.4 

27.3 

66 

219 

144  7 

1111 

1246 

22 

ISI 

48 

AFWT 

4.13 

4.78 

.4 

17.9 

72 

334 

1447 

1111 

12  46 

22 

ISl 

52 

ISIT 

3.66 

5.83 

.4 

26.6 

52 

43? 

1447 

nil 

12  46 

22 

ISI 

11 

STAN 

2.50 

3.53 

.4 

13.5 

55 

205 

1245 

1246 

1421 

2  2 

ISI 

48 

AFWT 

3.31 

4.51 

.4 

21.3 

32 

221 

1245 

1246 

1421 

22 

ISI 

52 

ISIT 

3.99 

4.93 

.4 

26.1 

75 

479 

1245 

1246 

1421 

22 

ISl 

11 

STAN 

3.58 

5  .40 

.4 

26.2 

50 

223 

1273 

1421 

1628 

2  2 

ISI 

48 

AFWT 

4.71 

5.94 

.4 

25.3 

34 

86 

1273 

1421 

16  28 

2  2 

ISI 

52 

ISIT 

3.55 

5.36 

.4 

26.3 

76 

581 

1273 

1421 

162« 

22 

ISI 

11 

STAN 

3.27 

5.43 

.4 

34.0 

86 

219 

1387 

1628 

1853 

22 

ISI 

48 

AFWT 

3.50 

4.85 

.4 

27.3 

97 

141 

1387 

1628 

1853 

22 

ISI 

52 

ISIT 

2.28 

3.28 

.4 

17.0 

54 

585 

1387 

1628 

1853 

22 

ISI 

11 

STAN 

2.37 

4.20 

.4 

30.3 

106 

174 

1517 

1853 

2325 

22 

ISI 

48 

AFWT 

2.99 

3.92 

.4 

17.8 

104 

149 

1517 

1853 

2325 

22 

ISI 

52 

ISIT 

4.42 

6.06 

.4 

26.4 

95 

743 

1517 

1853 

2325 

22 

ISI 

11 

STAN 

3.08 

4.15 

.4 

20.5 

91 

143 

823 

2325 

052° 

22 

ISl 

48 

AFWT 

3.29 

4.36 

.4 

25.7 

73 

94 

823 

2325 

052') 

22 

ISI 

52 

ISIT 

3.15 

4.18 

.4 

20. C 

97 

379 

823 

2325 

0529 

Table  ■'-.3,  Variation  of  processing  delay  with  time  of  day 

(516  IMP) 
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From 

To 

Mean 

SDev 

Min 

Max 

Smp  1 

To-Ch 

To-Task  Time 

Node 

Node 

Size 

Tot 

Tot 

Begin  End 

27 

BELV 

14 

CMli 

7.07 

7.27 

.7 

29. P 

140 

153 

341 

095? 

1111 

27 

BELV 

26 

SDAC 

7.71 

7.37 

.7 

25. P 

81 

81 

341 

0952 

1111 

27 

BELV 

29 

ABRD 

8.66 

8.72 

.7 

40.7 

98 

101 

341 

0952 

1111 

27 

BELV 

14 

CMU 

10.9 

11.0 

.7 

49.7 

165 

175 

431 

1111 

1246 

27 

BELV 

26 

S  OAC 

11.5 

11.5 

.7 

38. S 

114 

114 

431 

1111 

1246 

27 

BELV 

29 

ABRD 

VJ.3 

10.8 

.7 

42.7 

1  30 

131 

431 

1111 

1246 

27 

BELV 

1  4 

CMU 

1C. 2 

11.5 

.7 

57. H 

214 

231 

485 

1246 

1421 

27 

BELV 

26 

SDAC 

12.7 

13.1 

.7 

52.8 

92 

92 

485 

1246 

1421 

27 

BELV 

29 

ABRD 

1  2.6 

13.2 

.7 

51  .9 

153 

155 

485 

1246 

1421 

27 

BELV 

14 

CMU 

9.24 

9.61 

.7 

45.1 

1  28 

132 

256 

1421 

1628 

27 

BELV 

26 

SDAC 

6.94 

7.8? 

.7 

?2.1 

20 

2  0 

256 

1421 

1623 

27 

BELV 

29 

ABRD 

7.28 

7.90 

.7 

29.9 

102 

104 

256 

1421 

1623 

27 

BELV 

14 

CMU 

1U.0 

9.69 

.7 

51.7 

■j  ftp 

194 

402 

1628 

1853 

27 

BELV 

26 

SDAC 

5.09 

5.53 

.7 

16.0 

17 

17 

40? 

1628 

1853 

27 

BELV 

29 

ABRD 

7.84 

7.93 

.7 

34.7 

1  87 

187 

402 

1628 

1853 

27 

BELV 

14 

CMU 

9.61 

9.1 1 

.7 

40.6 

166 

168 

356 

1853 

2325 

27 

BELV 

26 

SDAC 

12.6 

13.2 

.7 

49.4 

27 

27 

356 

1853 

2325 

27 

BELV 

29 

ABRD 

8.38 

8.84 

.7 

43.  C 

159 

161 

356 

1853 

2325 

27 

BELV 

14 

CMU 

10.9 

9.97 

.7 

41.0 

246 

246 

520 

2325 

0529 

27 

BELV 

26 

SDAC 

5  .60 

6.30 

.7 

21  .0 

40 

40 

520 

2325 

0529 

27 

BELV 

29 

ABRD 

8.73 

9.83 

.7 

45.7 

232 

234 

520 

^325 

0520 

Table  3,4.  Variation  of  processing  delay  with  time  of  day 

(316  IMP) 


In  the  previously  described  exoeriment  it  was  not  pos¬ 
sible  to  empirically  prove  that  the  excessive  delay  is  not 
due  to  the  interference  of  packets^  since  sampling  was  done. 
To  establish  this/  we  resort  instead  to  another  experiment 
in  which  we  selected  two  nodes  and  collected  packet  trace 
with  a  frequency  of  one.  On  close  examination  of  the  data 
we  find  that  for  most  packets  having  a  long  processing  delay 
no  other  packet  overlaps  it  in  time.  We  note  that  while 
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some  overflows  occurred  (i.e.^  lost  data  due  to  heavy  traff¬ 
ic)  in  the  packet  trace  data  the  statistical  properties  are 
nearly  identical  to  those  of  the  sampled  data.  We  conclude 
that  most  of  the  time  the  excessive  delay  is  not  due  to  the 
interference  of  other  packets. 

At  first  glance/  these  long  processing  delays  seem 
rather  insignificant  in  the  entire  scheme  of  thinos.  Let's 
examine  them  a  bit  more  closely.  Table  3.5  gives  the  aver¬ 
ages  for  pro  cessing  delay/  waiting  time/  and  acknowledgement 
time/  for  three  packet  types  ~  (1)  subnet  control  (largely 
end-to-eno  acknowledgements/  i.e./  RFNMs)/  (?)  user  priori¬ 
ty/  and  (3)  non-priority  packets.  Our  interest  is  with  the 
store-and-f orward  delays/  out  we  list  the  reassembly  infor¬ 
mation  for  completeness.  The  sample  sizes  listed  consist  of 
two  numbers.  The  first  number  indicates  the  sample  size  for 
the  Channel  Waiting  and  Acknowledgement  statistics/  and  the 
second  refers  to  the  Processing  statistics.  Notice  that 
the  time  spent  waiting  for  and  being  served  by  the  Task 
routine  is  on  the  same  order  as  the  time  spent  waiting  for 
the  channel  for  the  516  IMP  and  always  greater  (about  dou¬ 
ble)  in  the  case  of  the  316  IMP.  We  note  that  for  these 
measurements/  non-priority  packets  traversed  a  316  node  fas¬ 
ter/  on  the  average/  than  both  control  packets  and  priority 
packets  (the  order  of  service  by  Task  is  FCFS)  even  though 
the  channel  waiting  time  is  greater  (as  one  would  expect) 
for  non-priority  packets!  Examination  of  more  extensive 
data  reveals  that  ID  of  36  cases  exhibit  this  behavior.  In 
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fact  in  only  16  of  58  cases#  including  all  node  types#  was 
processing  delay  smaller  for  both  priority  and  control  pack¬ 
ets.  The  experiments  which  we  have  conducted  in  trars^it- 
ting  simulated  speech  data  from  the  UCLA  POP  11/A5  Chayt  .A] 
as  well  as  similar  exoeriments  conducted  by  the  University 
of  Southern  California  Information  Sciences  Institute#  f*ari- 
na  Del  Rey#  California  (ISI)  CCohe  7A3  support  this  observa¬ 
tion.  In  those  experiments  no  discernible  difference  in 
delay  could  be  found  between  priority  and  non-priority  mes¬ 
sages!  In  a  report  from  Network  Analysis  Corporation#  Glen 
Cove#  New  York  (NAC)  [MAC  753  the  authors  state  that  s  AO'' 
decrease  in  mean  round-trip  delay  can  be  achieved  by  reduc¬ 
ing  the  processing  delay  to  its  original  estimate  of  1  msec. 
Our  own  projections  (which  appear  in  Section  3.3.5)  show 
that  a  AAX  decrease  could  be  achieved.  Therefore  rroc-'.  ss  i  no 
oelay  currently  accounts  for  at  least  2/5  of  the  total  de¬ 
lay#  which  is  quite  significant. 
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Store-and- Forward 

Reassembly 

Mean  SOev 

Mean 

SDev 

Control  Packets 

Sample  Size 

(  3790/  1116) 

(  0/ 

0) 

Process ing 

3.81  5.? 2 

0.0 

0.0 

Channel  Waiting 

2.31  10.40 

0.0 

n.o 

Acknowledgement 

19.11  15.56 

0.0 

c.o 

User  Priority  Packets 

Sample  Size 

(  731/  317) 

(  2098/ 

2C66) 

Process ing 

2.57  3.83 

3.36 

4.69 

Ch  anne 1  Waiting 

2.69  14.41 

5.87 

109.68 

A  c  know  1 edgement 

22.65  15.03 

4.36 

18.75 

User  Non-priority 

Packet  s 

Sample  Size 

(  1910,  260) 

(  567/ 

468) 

Process ing 

3. IP  4.95 

8.27 

27.60 

Channel  Waiting 

3.88  26.64 

12.20 

34.87 

Acknowledgement 

22.93  16.87 

6.1? 

11.0% 

Table  3.5(a).  Processing  delay  statistics 

(node  22  a  516  IMP/  14 

MAR  75) 

Store-and- 

-Forward 

Reassembly 

Mean 

SDev 

Mean 

SDev 

Control  Packets 

Sample  Size 

(  1546/ 

1507) 

(  0/ 

0) 

Processing 

10.43 

10.30 

0.0 

D.O 

Channel  Waiting 

4.89 

8.80 

0.0 

n.o 

Acknowledgement 

2C.73 

11.16 

0.0 

c.c 

User  Priority  Packets 

Sample  Size 

(  575/ 

575) 

(  11/ 

11 ) 

Processing 

9,63 

10.36 

14.82 

1G.7P 

Channel  Waiting 

5.00 

8.60 

0.36 

0.5  0 

A  cknow 1 edgement 

23.12 

10.94 

3.64 

5.43 

User  Non-priority 

Packets 

Sample  Size 

(  643/ 

617) 

(  16/ 

16) 

Process inc 

7.63 

9.08 

16.38 

22.62 

Channel  Waiting 

5.86 

12.68 

1 .56 

3.35 

Acknowledgement 

26.22 

12.61 

5.13 

5.06 

Table  3. Sib).  Processing  delay  statistics 

(node  27  a  316  IMP/  14  MAR  75) 
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Store-and-Forward 

Reassembly 

Mean  SDev 

Mean 

SDev 

Control  Packets 

Samole  Size 

(  7777,  2738) 

(  0^ 

0) 

Process ing 

3.75  4.7>^ 

0.0 

0.0 

Channel  Waiting 

2.72  15.23 

0.0 

r.o 

Acknowledgement 

21.43  13.04 

0.0 

0.0 

User  Priority  Packets 

Sample  Size 

(  1328x  423) 

(  4344/ 

4326) 

Processing 

2.92  4.12 

4.14 

4.9S 

Channel  waiting 

3.07  22.03 

0.34 

0.61 

A  cknow 1 edgement 

24.16  14.37 

2 .50 

0.97 

User  Non-priority 

Pa cket  s 

Sample  Size 

(  48S4>.  2356) 

(  613/ 

611 ) 

Pro  cess ino 

3.42  4.64 

4.81 

5.44 

Channel  Waiting 

5.01  17.04 

0.44 

0.97 

Acknowledgement 

27.20  13.66 

4.58 

7.37 

Table  3.5(c).  Processing  delay  statistics 

(node  22  a  516  IMP^  24 

JUL  75) 

Stor  e-and' 

-Forward 

Reassemb  ly 

Me  an 

SDev 

Mean 

SDev 

Control  Packets 

Sample  Size 

(  4317/ 

4258) 

(  0/ 

0) 

Pro  cess ing 

8.1? 

8.39 

0.0 

0.0 

Channel  Waiting 

3.33 

6.63 

0.0 

0.0 

Acknowledgement 

19.74 

11.33 

0  .0 

c.c 

user  Priority  Packets 

Sample  Size 

(  1  543/ 

1543) 

(  4/ 

4) 

Processing 

8.47 

8.16 

3.75 

4.86 

Channel  Waiting 

3.56 

7.79 

0.75 

0.5C 

Acknowledgement 

22.24 

10.52 

2.50 

1  .77 

User  Non-priority 

Packet  s 

Sample  Size 

(  1737/ 

1700) 

(  3/ 

3) 

Process ing 

6.53 

7.85 

8.33 

7.02 

Channel  Waiting 

4.77 

8.60 

1  .00 

0.0 

Acknowl edgement 

23.42 

10.22 

1  .00 

0.0 

Table  3. Sid).  Processino  delay  statistics 

(node  27  a  316  inP^  24  JUL  75) 


The  acknowledqement  time  is  larger  than  one  r.iaht  ex¬ 
pect.  This  can  have  only  a  second  order  effect  on  delay. 
If  packets  are  retained  longer  than  necessary  when  buffers 
are  in  short  supply  then  some  packets  may  have  to  be 
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retransmitted.  This  is  not  occurring^  for  if  it  were  the 
effect  would  be  recorded  as  waiting  time  since  retransmis¬ 
sions  change  the  "sent  time"  in  the  trace  block. 

3.3  Analysis 

To  pinpoint  the  cause  of  the  excessive  processing  delay 
we  shall  use  a  model  of  the  nodal  priority  structure.  A 
description  of  that  structure  follows. 

3.3.1  S^sigm  dgsctiatiaQ 

The  IMFs  are  required  to  perform  a  set  of  functions  in 
order  that  the  network  operate  smoothly.  Among  these  func¬ 
tions  are:  receiving/  routing  and  sending  packets;  process¬ 
ing  routina  updates  and  periodically  sending  routing  updates 
to  neighbor  nodes.  A  priority  level  is  assigned  to  each  of 
the  various  functions  within  the  IMP.  A  partial  list  cf  the 
levels  appears  in  Table  3.6.  A  function  is  activated  by  an 
interrupt  mechanism.  Only  the  function  associated  with  the 
highest  active  priority  level  may  occupy  tbe  processor  at  a 
given  instant. 
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Abbreviation  Level  Meaning^  Comments 

N2I  (highest)  0  Modem  to  IMP^  Channel  input 

I?M  2  IMP  to  Modern^  Channel  output 

I2H  3  IMP  to  HOST/.  HOST  output 

H21  4  HOST  to  IMP/  HOST  input 

T.O  S  Timeout/  Periodic  functions 

TSK  6  Task/  Primarily  packet  routing 

8CK  (lowest)  7  Background/  Statistics/  etc. 

Table  3.6.  IMP  priority  levels 

The  portion  of  the  priority  structure  in  which  we  are 
most  interested  here  is  that  governing  the  processing  of 
routing  update  packets  and  other  ( non- rout i n g )  packets. 
Onoer  the  current  scheme/  pictured  in  Fioure  3.1(a)/  all 
packets  arrive  and  are  treated  by  the  input  routine  running 
at  the  M2 1  level  which  places  them/  without  examination  as 
to  type  or  oriority/  on  the  Task  queue.  They  are  then  pro¬ 
cessed  in  f  i  rst -come-f  i  rst-serverf  order.  When  the  Task 
routine  encounters  a  routing  packet  it/  in  effect/  switches 
oriority  to  level  T.o  so  as  not  to  be  interrupted  by  any 
function  of  equal  or  lower  priority.  (This  has  the  effect 
of  disallowing  the  sending  of  a  routing  update  packet  to  a 
neighbor  while  the  table  is  in  the  process  of  being  undat¬ 
ed.)  Hon-routing  packets  are/  on  the  other  hand/  interrupt- 
able  by  T.O  functions.  More  detailed  descriptions  appear  in 
CCole  71]  and  in  CMcQu  72]. 
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Figure  3.1(a).  Simplified  model  of  ARPANET  queue  structure. 
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Figure  3.1(b).  Simplified  model  of  ARPANET  queue  structure  with  priority  within  the  TASK  queue. 


Since^  as  we  see  later#  routing  update  packets  require 
much  processing  compared  with  non -routing  packets#  it  seems 
appropriate  to  reorder  the  priorities  and  treat  routin'’ 
update  packets  in  the  PC<  level  and  other  packets  at  the  T?k 
level.  Figure  3.1<b)  shows  the  logical  structure  of  such  a 
system.  Notice  that  the  Task  queue  has  two  levels  of  prior¬ 
ity  instead  of  one  as  before.  Also  notice  that  unlike  the 
channel#  the  Task  routine  serves  the  queue  by  preempting#  if 
necessary#  the  (low  priority)  routing  updates.  T^-is  scheme 
is  not  new.  Originally  routing  packets  were  treated  in  the 
tiCk  level.  The  cnief  reason  for  the  current  set  of  priori¬ 
ties  coupled  with  higher  frequency  updating  is  to  propaoate 
routing  information  faster  through  the  larger  network 
rsant  75j. 

3.3.2  General  thegc^ 

This  system  can  be  modeled  by  a  single  server  head  of 
the  line  (HCL)  preemptive  priority  queueing  system 
LKlei  763.  Tne  solution  aiven  in  [klei  763  for  the  mean 
time  in  system  for  priority  class  p  is  valid  under  the  ?*/G/1 
assumption  (which  we  later  adopt).  There  are#  however#  some 
partial  results  for  the  G/G/1  case  which  we  may  apply  here. 
Schrage  CSchr  oS3  proves  that  the  shortest  remaining  pro¬ 
cessing  time  first  (SRFT)  seneduling  algorithm  is  optimal  in 
that  it  minimizes  the  average  number  of  jobs  in  the  system. 
That  is#  the  number  of  jobs  in  a  system  unde,  the  SRPT  algo¬ 
rithm  is  less  than  or  equal  to  the  number  in  the  system 
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under  any  other  scheduling  algorithu  for  the  same  seouence 
of  jobs.  Kleinrock  [Klei  76]  points  out  that  if  the  cost  of 
delaying  a  customer  is  constant  over  all  priority  classes/’ 
then  the  SRH  rule  achieves  the  minimum  cost.  f'ence/  the 
theoretical  results  indicate  that  an  arrival  should  join  the 
queue  behind  all  those  customers  in  the  system  with  remain- 
ina  service  time  which  is  less  than  (or  equal  to)  the  ser¬ 
vice  time  required  of  that  arrivals  and  in  front  o*  all  oth¬ 
ers  in  the  system.  That  is  all  non-routino  packets  should 
preempts  if  necessary#  the  processing  of  routing  packets# 
since  as  we  see  below#  routing  packets  require  much  more 
processing  than  do  non-routing  packets. 

havino  stated  the  general  theoretical  result  we  now 
wish  to  show  the  order  of  magnitude  of  the  improvement  one 
may  achieve  by  causing  routing  updates  to  be  processed  at  a 
lower  priority  than  other  packets. 

3.3.3  5  mgsJsL 

In  the  following  analysis  we  wish  to  examine  the  tra¬ 
deoff  between  the  existina  and  the  oroposed  system  over  the 
full  range  of  packet  traffic  intensity.  Some  simplifying 
assumptions  will  allow  the  direct  application  of  the  theory. 
For  simplicity#  we  shall  lump  all  high  priority  functions 
into  the  M?1  priority  class.  We  consider  two  such  high 
priority  functions  -  (a)  modem  input  and  (b)  modem  output^ 
and  ignore  all  others.  We  shall  overestimate  the  time  used 
in  these  functions  which  will  tend  to  give  a  lower  bound  on 
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perfornance  (i.e./^  a  pessimistic  value).  We  assume  that  the 
arrival  of  customers  to  each  class  is  independent  of  the 
arrivals  to  the  other  classes  and  may  be  modeled  by  a  Pois¬ 
son  process.  The  Poisson  assumption  on  the  arrival  of  pack¬ 
ets  is  equivalent  to  assuming  that  we  have  a  large  number  of 
input  channels  collectively  sending  at  the  same  rate  as  the 
actual  number  of  channels.  We  further  assume  that  each 
queue  has  no  restriction  as  to  its  length  (i.e./  infinite 
nodal  storage)  and  that  the  overhead  for  changing  tasks  is 
negligible.  The  current  system  then/  may  be  modeled  by  a 
two  level  preemptive  resume  HOL  priority  queueing  system  and 
the  proposed  system  by  a  three  level  system.  These  systems 
are  pictured  in  Figure  3.2.  In  each  system  the  input/output 
function  has  oreemptive  priority  over  the  other  classCes). 
T^e  three  level  system  (in  part  <b)  of  the  figure)  divides 
the  Task  queue  arrivals  into  routing  updates  and  non -routing 
packets/  the  latter  having  preemptive  priority  over  the 
former . 
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3.3.4  Anai.3tti£  cgsuits 

The  general  solution  from  CKlei  761  for  the  average 


time  in  system  for  a  customer  of  the  pth  class  is 

P 

x(p)[1-s(p)]  +  r<i>x?(i)/2 

j=P 

T{p,p)  =  -  (3.1) 

C1-s(p)3C1-s(p+1)J 

where  P  is  the  number  of  classes  (1  is  the  lowest  priority 
class  and  P  is  the  highest)/  xlp)  is  the  mean  service  time 
for  a  customer  in  class  p/  x^lp)  is  the  second  moment  of 
service  time  for  a  customer  in  class  p/  r(i)  is  the  input 
rate  of  customers  of  class  j/  and  sip)  is  the  cumulative 
utilization  due  to  classes  of  priority  p  and  higher  (where 
the  utilization  for  class  j  is  x(j)r(j)).  Let  x(i)  =  the 
service  time  for  the  M2I  class/  x (u)  =  the  service  time  for 
routing  update  processing/  x(n)  =  the  service  tine  for  non¬ 
routing  packets/  r(u)  =  the  arrival  rate  of  routing  update 
packets/  and  r(n)  =  the  arrival  rate  of  non-routing  packets. 
For  the  purpose  of  further  simplifying  the  model  we  shall 
assume  that  the  service  time  in  each  class  is  fixed.  In 
particular/  this  means  that  the  second  moment  is  equal  to 
twice  the  mean.  V.'e  assume  that  each  packet  processed  by  the 
Task  routine  arrives  and  eventually  leaves  via  the  M2l  level 
(since  each  packet  processed  by  the  Task  routine  must  arrive 
via  the  input  routine  and  exit  via  the  output  routine  in  our 
model).  we  also  assume  that  the  number  of  routing  update 
packets  received  by  a  node  is  equal  to  the  number  sent.  We 


therefore  assume  (an  independent  arrival  process  with)  an 
arrival  rate  of  2Cr(u)+r(n)]  to  the  M?i  class.  From  equa- 
t ion  3.1  we  have 


1(1, Z)  = 


x(u) r(u)+x (n) r(n) 


[r<u)+r(n)DC1-2x(i)Cr(u)*r(n)D] 


t  2  2 

Lx  (u)r(u)+x  (n)r(p)1/2  +  Cr(u)+r(n)]x  (i) 

♦  "  ■  —  ■  - ,  III  -  -  -  '  .  —  .  ■■  ■  —  ■  ■  - 

C1-x(u) r(u)-x(n)r(n)-2x<i) Cr(u)+r(n>D3C1-2x(i) Cr(u)+r(n) ]J 


(3.2) 


Equation  3.2  oives  the  average  time  in  systam  fcr  routing 
and  non-routing  packets  in  the  two  level  system.  The  solu¬ 
tion  to  the  three  level  system  is 

x(u) 


7(1,3)  =  - 

1-x(n)r(n)-2x(i)Cr(u)'‘-r(n)D 

2  2  ? 
r(u)x  (u)/2  +  r(n)x  (n)/2  +  Cr(u)+r(n)3x  (i) 


1-x(n)r(n)-2x(i)Cr(u)+r(n)3 


1-x(u)r(u)-x<n)r(n)-2x(i)Cr(u)+r(n)D 


(3.3) 


T(2,3) 


X  (n) 


1-2x(i)Cr(u)+r(n)3 


2  ? 

r(n)x  (n)/2  +  tr(u)+r(n)Dx  (i) 

+  ■ 

ri-x(n) r(n)-2x (i) Cr(u)+r(n) D:C1-2x(i) Cr(u)+r(n) 33 


(3.4) 
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Since  both  systems  are  preemptive^  there  is  no  difference 
between  T(2^2)  and  T(3^3).  Therefore  we  need  only  concern 
ourselves  with  T(?r3)  and  T(l,3).  T<U2)  -  J(2,3) 
gives  the  reauction  in  delay  for  non-routing  packets  when 
changing  from  the  two  level  system  to  the  three  level  sys¬ 
tem.  T(1x3)  -  T(1r2)  gives  the  increase  for  routing  update 
processina  for  the  same  change.  Comparison  of  the  two  sys¬ 
tems  depends  on  the  relative  values  of  x(u)^  x(n)/  r(u)  and 
r(n)  which  we  shall  now  estimate  in  order  to  oresent  some 
numerical  results- 


3.3.5  Numerisai,  cgsyitS 

We  estimate  that  no  more  than  500  machine  cycles  are 
required  to  process  an  incoming  packet  or  an  outcoino  pack¬ 
et.  we  further  assume  that  a  maximum  of  300  cycles  are 
required  to  do  the  TSK  level  processing  of  a  non-rcuting 
packet.  These  values  are  ourposely  larger  than  the  estimate 
in  CKcdu  721.  This  is  in  order  to  exaggerate  their  effect 
in  the  model  and  therefore  provide  a  pessimistic  projection 
of  performance  (as  mentioned  earlier).  Pouting  packets 
require  much  more  time  to  digest.  In  a  recent  experiment 
carried  out  BBN  CB6N  75j  with  a  316  IMP#  it  was  found  that 
18'a  of  the  HOST  througnput  was  lost  when  going  from  2ero  to 
one  50kbps  line.  A  similar  loss  was  noted  with  each  addi¬ 
tional  line.  Assuming  that  the  processor  was  fully  utilizer* 
durino  the  experiment/  this  indicates  that  routing  and  other 
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processes  associated  with  IMP-to-IMP  tines  require  14400 
cycles/routing  update  period.  The  time  could  be  signifi¬ 
cantly  reduced  by  implementing  the  "local  loop"  prevention 
algorithm  outlined  in  INayl  75D  and  chapter  2.  That  isr 
many  (one  third  on  the  average  for  3-connected  neighbors)  of 
the  routing  update  entries  are  skipped  since  they  point 
toward  the  updating  node.  We  estimate  those  orocesses  in¬ 
volved  in  handling  one  line  (with  no  data  traffic)  as  fol¬ 
lows: 


Function 


Cycles /update  period 


Receive  routing  messaoe 

Input  (140  on  316)  2}<0 

M2I  300 

TSk  300 

Receive  and  process  IHY  100 

Send  routing  packet  300 

T.O  300 

12M  300 

Output  (140  on  316)  2«C 

Process  routing  packet  12C00 

Other  540 


Total  14400 


Table  3.7  Channel  function  processing  time  estimates 


#(e  shall  use  for  x(u)  a  value  of  12000^  which  is  a  con¬ 
servative  estimate  to  minimize  the  effect  of  routing.  This 
conflicts  with  the  pessimistic  estimates  of  inputs  outout 
and  non-routing  packet  processing  and  would  tend  to  give  an 
optimistic  value  of  performance  (i.e.»  lower  than  actual 
delay).  However^  it  does  result  in  a  lower  bound  (or  pes¬ 
simistic  estimate)  for  the  improvement  in  performance 
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between  the  two  systems.  The  service  time  of  the  various 
tasks  is  a  function  of  the  processor  speed.  In  August  1976 
there  were  four  processor  types  in  the  ARPANFT  —  the  516 
IMP#  316  IMP#  316  TIP#  and  Pluribus  [Hear  763  IMF  according 
to  CNIC  763.  We  shall  consider  only  the  first  three  which 
have  speeds  of  .96#  1.6  and  2.2  usec/cycle  respectively 
CMcQu  733.  The  rate  of  arrival  of  routing  packets  to  a  node 
is  governed  both  by  the  number  of  connected  channels#  and 
the  utilization  of  those  channels  as  mentioned  in  chapter  2. 
The  three  nodal  types  were  distributed  with  connectivity 
CNIC  763  as  shown  in  Table  3.*<.  Also  shown  in  Table  3.F  is 
the  predicted  "zero-load"  average  waiting  time  expressed  in 
milliseconds  for  each  case  (i.e.#  the  waiting  time  when  r(n) 
=  0).  each  pair  in  the  table  consists  of  the  number  of 
nodes  in  the  network  with  this  connectivity  followed  by  the 
zero  load  delay  prediction.  At  that  time  there  was  one 
five-connected  Pluribus  IMP  in  the  network. 

Connectivity  5‘*6  516  TIP 


1 

0# 

.63 

2# 

1 .87 

3# 

3.79 

2 

5# 

1  .40 

9# 

4.57 

12# 

1D.3» 

3 

11# 

2.36 

5# 

8.83 

9# 

24.93 

4 

1# 

3.61 

1# 

16.66 

0# 

86.17 

Table  3.?  Connectivity  of  processor  types 

The  zero  load  average  waiting  time  for  non-routing  packets 
in  the  three  level  system  is  essentially  zero.  The  range  is 
from  .000645  to  .0130  msec.  Comparing  these  values  with 
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those  in  Table  3.8  we  see  the  drafflatic  effect  of  this  change 
in  priorities.  The  zero  Load  average  waiting  time  for  non¬ 
routing  packets  in  the  two  level  system  is  from  about  1000 
to  6000  times  what  it  is  in  the  three  level  system! 

The  behavior  of  these  systems  over  the  complete  range 
of  channel  utilization  (where  packets  of  300  bits  average 
lengtn  occupy  the  channels)  is  shown  in  Figure  3.3  parts  (a) 
through  (f).  Also  shown  in  the  figure  are  measured  data 
points  (as  squares)  where  such  data  exists.  We  show  only 
the  results  for  the  two  and  three  connected  nodes  (those 
which  are  most  prevalent  in  the  ARPANET).  Each  part  of  Fig¬ 
ure  3.3  consists  of  three  curves.  The  curve  which  lies 
between  the  two  others  shows  the  performance  of  the  two  lev¬ 
el  system  and  the  other  two  show  the  three  level  system's 
performance.  Average  waiting  time  for  routing  and  non¬ 
routing  packets  is  shown  in  the  highest  and  lowest  curves 
respectively.  The  gain  in  delay  for  non-routing  packets  is 
the  difference  between  the  middle  and  the  lowest  curve.  The 
cost  in  delay  for  routing  is  the  difference  between  the 
hig'est  and  the  middle  curve.  For  parts  (a)  and  (b)  the 
bottom  curve  is  almost  indistinguishable  from  the  axis 
(i.e.^  nearly  zero  waiting  time  through  the  entire  range). 
Notice  in  parts  (b)  and  (d)  that  the  model  is  a  good  lower 
bound  to  the  measured  results.  Since  we  overestimated  the 
high  level  processing  time«’  an  upper  bound  was  expected. 
The  fact  that  a  lower  bound  was  achieved  is  most  likely  due 
to  the  underestimation  of  x(u). 
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Figure  3.3.  (b)  Mean  waiting  time  on  TASK  queue  (3-connected  516  IMP). 
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Figure  3.3.  (d)  Mean  waiting  time  on  TASK  queue  (3-connected  316  IMP). 
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Figure  3.3.  (f)  Mean  waiting  time  on  TASK  queue  (3-connected  TIP). 
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Substituting  the  two  level  syste*  model  for  the  con¬ 
stant  processing  delay  term  in  the  CKlei  74]  model  causes 
the  predicted  delay  to  increase  to  165  msec  for  ^arch  1977. 
This  is  an  increase  of  ^26X  (i.e.^  from  73  to  165  msec)  over 
the  August  1973  value  of  predicted  delay.  Recall  that  the 
actual  increase  in  delay  was  145X  (i.e.#  form  93  to  228 
msec).  A  better  (i.e./  larger)  estimate  for  x(u)  would 
drive  this  prediction  closer  to  the  actual  delay.  Assuming 
tnat  the  ratio  of  predicted  delay  to  actual  (measured)  delay 
remains  fixed  (i.e.#  73/93)#  then  the  projected  mean  round- 
trip  delay  for  March  1977  would  be  approximately  129  msec  if 
routing  updates  were  processed  at  tow  priority.  This 
represents  a  441!  reduction  in  delay! 

It  is  clear  from  these  results  that  a  substantial 
reduction  in  nodal  processing  delay  would  result  from  reord¬ 
ering  the  priorities  within  the  system.  But  this  is  a  local 
optimization  which  may  eventually  lead  to  worse  overall  pe  • — 
formance  in  a  network-wide  sense.  Because  the  speed  of  the 
processing  cf  routing  information  is  substantially  reduced 
by  the  reordering#  it  may  happen  that  routing  information 
will  rot  be  propagated  as  quickly  as  needed.  Looking  at  the 
three-connected  TIP  for  example#  we  see  that  routine  pro¬ 
cessing  incurs  infinite  delay  at  about  .6  channel  utiliza¬ 
tion.  To  determine  whether  this  local  optimization  leads  to 
a  global  optimum  or  leads  instead  to  disaster#  let  us  exam¬ 
ine  the  simulation  results  in  the  next  section. 
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3. A  UluiallQQ  ££SUUS 


A  simulation  was  performed  under  the  same  conditions  as 
described  in  chapter  2.  The  local  loop  free  algorithir  was 
modified  to  perform  routing  processing  at  low  priority. 
This  new  algorithm  (LLFLR)  was  used  in  the  simulation.  The 
result  is  compared  in  Figure  3.4  to  the  LLFR  algorithm  of 
chapter  2.  The  figure  shows  the  network-wide  mean  delay  as 
a  function  of  offered  load.  It  appears  that  the  local  op¬ 
timization  in  this  case  leads  to  a  global  optimum/  since 
LLFLR  out  performs  LLFR  at  each  load.  One  would  therefore 
expect  that  if  low  priority  routing  uodate  processing  were 
done  in  the  ARPANET/  then  the  mean  round-trip  delay  would 
fall  much  closer  to  the  predicted  (rather  than  the  measured) 
values  in  Table  3.1  for  lay  1974  and  August  1977/  as  indi¬ 
cated  in  the  last  section.  This  would  reduce  the  mean  delay 
to  be  within  the  specification  of  2D0  msec. 
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Figura  3.4.  High  and  low  priority  routing  (simulation). 
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3.5  CQQ£iuslQQS 


There  are  two  important  general  lessons  to  be  learned 
here.  (1)  In  order  to  provide  different  levels  of  service 
to  different  classes  of  customers#  one  must  be  careful  to 
provide  that  order  of  service  at  each  step  of  the  process¬ 
ing.  For  example#  if  one  wishes  to  guarantee  that  priority 
packets  traverse  the  network  more  quickly#  on  the  average# 
than  do  non-priority  packets#  then  priority  packets  must  be 
serviced  with  priority  from  queues  in  w^'ich  they  may 
wait  (i.e.#  the  Task  queue  as  well  a;  the  channel  Queue#  in 
this  case).  id.)  Examination  of  the  worst  case  phenomenon 
should  not  be  used  alone  in  arriving  at  decisions  regarding 
priority  of  service.  Rather  one  should  evaluate  the  impli¬ 
cation  of  those  decisions  over  the  entire  range  of  system 
oper  at  ion . 

We  have  cone  ent  r  atecf  in  this  chapter  on  a  small  (but 
important)  part  of  the  priority  structure  within  the  ARPANET 
nodes.  There  are  other  areas  within  the  system  priority 
structure  which  may  bear  fruit  in  terms  of  reducing  delay  or 
increasing  throughput.  There  is  a  oriority  structure  with 
which  the  channels  are  accessed  as  well#  which  provides 
another  area  for  investigation. 

In  this  chapter  a  high  degree  of  overhead  due  to  the 
ARPANET  routing  update  procedure  was  exposed.  The  next 
chapter  concentrates  on  the  issue  of  overhead  in  periodic 
adaptive  routing  in  large  networks. 
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CHAPTER  4 


ON  THE  EFFECT  OF  PERIODIC  UPDATE  ROUTING  PROCEDURES 


iDt£2dy£liCQ 

In  a  packet  switched  network#  some  form  of  adaptive 
routinq  procedure  is  desirable  so  that  packets  may  be  routed 
around  line  and  node  failures  and  possibly  around  congestion 
in  the  network  and  to  allow  the  network  to  adjust  to  dras¬ 
tics  changes  in  input  traffic  matrix.  It  is  clear  that 
there  must  be  some  overhead  associated  with  any  form  of 
adaptive  routing  (i.e,/  the  channel  time  and  processor  time 
required  to  generate#  transmit#  and  process  the  routino 
information).  Clearly#  one  would  hope  that  the  cost  for 
such  adaptive  routing  does  not  exceed  the  benefits  derived 
therefrom.  Since  adaptive  routing  is  considered  to  be 
necessary  in  practice#  its  overhead  has  received  only  par¬ 
tial  consideration  by  most  authors  CCegr  753#  CFult  723# 
EMcCo  753#  [Mc&u  743#  CPick  763.  In  this  chapter#  we  study 
some  unusual  phenomena  caused  by  the  interference  of  routinq 
updates.  Specifically#  we  consider  the  cost  (in  terms  of 
message  delay)  of  the  current  ARPANET  routing  update  pro¬ 
cedure.  we  begin  by  presenting  some  results  of  a  set  of 
measurement  experiments  which  prompted  an  analysis  of  the 


effects  of  the  routing  procedure  on  message  deley.  A  sim¬ 
plified  model  of  the  system  is  then  discussed  and  analyzed 
exactly.  This  exact  solution  is  a  bit  unwieldy  for  highly 
detailed  models  in  which  case  we  resort  to  simulation  to 
show  the  performance  and  to  demonstrate  the  effect  of  modi¬ 
fied  routing  schemes.  The  simulation  results  indicate  a 
rather  high  cost  associated  with  the  use  of  periodic  routing 
updates  in  networks  the  size  of  the  ARPANET.  This  suggests 
the  use  of  a  "passive"  routing  scheme  with  catastrophe- 
triggered  updates. 


A. 2  ??gasyrgn!£Qi 

A/e  set  out  to  determine  by  what  means  and  how  accurate¬ 
ly  one  could  predict  round-trip  network  delay  for  a  stream 
of  messages  with  fixed  interarrival  times  based  on  previous 
delay  samples  in  the  ARPANET.  This  traffic  pattern  is  exhi¬ 
bited  by  fixed  data  rate  sources  such  as  speech  CForg  75aD. 
These  measurement  experiments  were  conducted  with  no  inten¬ 
tion  of  considering  the  effects  of  the  periodic  update 
scheme  used  in  the  ARPANET.  We  observed  a  much  lower  than 
expected  correlation  between  successive  delays  (see  Chapter 
5).  In  experiments  sending  data  as  fast  as  possible/  suc¬ 
cessive  delays  display  higher  correlation  CKlei  7Sa]  than  do 
those  for  a  fixed  interarrival  time  source.  A  closer  look 
(suggested  by  D.  Cohen  of  the  Information  Sciences 
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Institute/  University  of  Southern  California)  revealed  some 
interesting  phenomena  regarding  the  effect  of  periodic  rout¬ 
ing  update  procedures. 

The  experiments  took  place  on  Friday  evening  December 
12/  1975/  between  the  hours  of  9  and  11  p.m.  PST.  (A  light 

network  load  and  therefore  nearly  constant  delay  was  expect¬ 
ed  during  this  time  period.)  Full  single-packet  messages 
were  sent  from  the  UCLA  PDP  11/45  to  a  "discard  fake  HOST" 
(a  portion  of  the  ARPANFT  I*iP  software  which  mimics  a  "real 
HOST"  acting  as  a  sink  for  a  message  stream/  see  CBBK  6°]) 
over  (minimum  hop  path)  distances  of  1/  ?/  5  and  10  hops  at 
fixed  interdeparture  times  of  124/  165/  and  24P  msec.  The 
round-trip  delay  <i.e./  the  delay  from  the  time  the  message 
is  ready  to  be  sent  until  the  end-to-end  acknowl edgement 
RFNM  —  is  returned)  was  measured  by  the  PDP  11/45  and 
recorded  for  subsequent  study. 

Figures  4.1  through  4.4  show  some  of  the  round-trip 
delay  measurements  plotted  against  message  sequence  number 
(i.e./  time).  Here  we  show  network  delay  as  a  function  of 
(message  arrival)  time.  These  particular  samples  are 
representative  of  the  collection  of  experiments. 

N'otice  the  unusual  increases  in  delay  at  regular  intet — 
vals  (of  about  3U  messages)  in  Figure  4.1.  In  each  interval 
there  is  a  group  of  three  dominant  oeaks  separated  by  two 
regularly  spaced  points  where  the  delay  is  near  the  minimum. 
Notice  also  the  regular  decrease  in  the  first  peak  in  each 
group  with  time  until  it  is  replaced  with  a  full  sized  peak 
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at  intervals  of  five  groups.  This  curve  is  in  fact  a 
periodic  function  (with  some  "noise"  due  to  the  "other" 
backgrouno  data  traffic)  with  a  period  of  about  15C  mes¬ 
sages!  (Our  model  in  a  later  section  and  the  correlation 
results  in  Chapter  5  show  that  the  period  is  actually  160.) 

This  periodic  behavior  is  less  noticeable  at  longer 
network  distances.  There  is  a  pattern  of  climbing  to  a 
local  maximum  and  suddenly  dropping  and  starting  the  climb 
again  in  Figures  4.2  and  4.3.  Generally  speaking  increases 
in  delay  are  gradual  while  decreases  are  immediate  (though 
the  opposite  condition  occasionally  occurs  as  well).  The 
curve  shown  in  Figure  4.4  varies  quite  severely.  There  is  a 
pattern  however^  and  very  close  examination  reveals  a 
periodic  function  (again  with  some  noise)  with  a  period  of 
approximately  130  messages.  Plots  of  other  samples  show 
that  the  shape  of  the  curves  is  more  related  to  the  data 
rate  than  to  network  distance. 

One  would  not  normally  expect  to  see  such  a  regular 
variation  in  delay  with  time  assuming  no  control  or.  the  oth¬ 
er  network  traffic.  Rather  one  might  expect  to  see  a  random 
function  of  time^  possibly  with  a  slowly  varying  average 
which  changes  with  network  load  CCohe  74].  (Indeed^  the 
latter  was  our  hope;  we  were  looking  for  delay  predictors.) 
Below  we  present  an  analysis  of  the  regularity  of  these 
delay  functions. 
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Figure  4.1.  Round-trip  delay  measurement  (1  hop,  s*124). 
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Figure  4.2.  Round-trip  delay  measurement  (10  hops,  s=124). 


89 


000 

KIO 

000 


OL . .  . .  .  ■  ■  ■  - 

0  SO  lXl$O2QOaoaOOaSO4QO40OMO0iO0ODiOO7OO7«OOOOaOOOOO96O 

MfttAOO  MJMMII 


Figure  4.3.  Round-trip  delay  measurement  (5  hops,  s=248). 
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Figure  4.4.  Round-trip  delay  measurement  (2  hope,  s■165). 
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Before  proceeding  with  the  analysis/  we  must  first 
describe  the  important  features  of  the  periodic  routing 
update  scheme  in  the  ARPANET. 

AEPfifiEI  BStiQdit  Lftytioa  schetng  de^ctitticD 
There  exists  a  basic  routing  interval  of  640  msec.  The 
beginning  time  for  the  basic  period  is  chosen  essentially  at 
random  for  each  half-duplex  channel  in  the  network.  It  was 
noted/  by  BBN/  that  as  the  network  grew  in  si7e/  routing 
information  was  not  being  propagated  in  a  timely  fashion 
with  only  one  update  per  basic  period  IBPK  743.  Therefore 
provision  was  made  to  send  up  to  five  updates  in  one  basic 
oeriod.  taring  a  basic  period  the  line  utilization  (includ¬ 
ing  that  for  updates)  is  measured  to  determine  the  number  of 
updates  to  be  sent  during  the  next  basic  period.  For  each 
additional  2CX  of  line  utilization/  one  of  the  five  possible 
updates  is  dropped.  For  example/  at  65T!  line  utilization 
only  two  updates  are  sent  in  the  next  basic  period.  It  is 
important  to  note  that  the  updates  are  not  necessarily  even¬ 
ly  spaced  within  the  basic  period.  Rather  this  period  is 
divided  into  five  equal  segments.  Routing  updates  are  sent 
only  at  segment  boundaries  (i.e./  every  128  msec).  For  the 
20  to  40%  range/  for  example/  updates  are  sent  at  C/  128/ 
384/  and  512  msec  into  the  basic  period  (rather  than  0/  160/ 
320/  480  msec  for  evenly  spaced  updates).  Routing  update 

packets  were  1160  bits  in  length/  requiring  23.2  msec  to 
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transmit  on  the  50  kbps  channels  used  in  the  ARPANET  (recent 
changes  have  increased  this  somewhat).  Additionally#  ap¬ 
proximately  12000  machine  cycles  (based  on  a  treasurement 
reported  in  CBBN  750#  see  Chapter  3)  are  required  to  cigest 
an  incoming  routing  update  (i.e.#  11,52  msec  for  a  516  IMP 
and  19.2  msec  for  a  316  IMP).  Chapter  3  examines  the  impor¬ 
tance  of  this  overhead  within  the  nodes. 

4.3.2  A  simele  rngjigl. 

Cur  analysis  uses  a  queueinc  system  with  some  "back¬ 
ground"  traffic  (i.e.#  routing  and  ambient  data  traffic)  to 
which  we  add  a  stream  of  deterministically  generated  traf¬ 
fic.  The  inclusion  of  the  ambient  data  traffic  is  to  model 
the  interference  caused  by  other  packet  sources  in  the  sys¬ 
tem.  We  wish  to  study  the  system  time  of  this  added  stream 
traffic  (i.e.#  the  round-trip  delay  as  shown  in  Figures  4.1 
through  4.4),  We  first  examine  the  waiting  time  on  e  single 
channel.  The  model  for  the  single  channel  is  pictured  in 
Figure  4.5.  There  are  three  classes  of  customers  arriving 
to  3  single  oueue#’  routing  update  packets  (R)#  ambient  data 


Figure  4.5.  Sin^  chaiNNl  model. 
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packets  (A)#  and  strean  packets  (S).  All  arrivals  are 
served  in  f i rst-cone-f i r st-served  (FCFS)  fashion  (thouqh  in 
the  actual  system^  routing  update  packets  take  precedence 
over  both  priority  data  packets  and  control  packets^  all  of 
which  take  precedence  over  non-priority  data  packets;  see 
Chapter  3).  The  effect  of  this  is  negligible  for  our  pur¬ 
pose  here  as  we  show  later. 

Let  w(t)  represent  the  waiting  time  for  a  stream  traf¬ 
fic  packet  arriving  at  time  t.  Suppose  U(t)  is  the  amount 
of  work  of  type  R  and  A  remaining  in  the  system  at  time  t. 
As  Long  as  the  stream  arrivals  have  no  priority  over  other 
customers/  an  arrival  must  wait  until  all  work  in  the  system 
has  been  completed  before  receiving  any  service  of  its  own. 
The  amount  of  work  in  the  system  found  by  an  arrival  at  time 
t  is  at  least  U(t).  Hence 

u(t)  >  U(t) 

Equality  is  achievea  for  an  arrival  when  it  is  the  first 
stream  traffic  arrival  in  a  busy  period  (i.e./  a  period  of 
continuous  activity  by  the  server). 

For  simplicity  of  the  following  analysis/  we  assume 
that  the  ambient  data  traffic  has  zero  intensity,  lift)  for 
a  system  void  of  any  data  traffic  is  shown  in  Figure  4.6. 
At  the  arrival  time  of  a  routine*  update  packet/  the  amount 
of  work  in  the  system  jumps  up  by  23.2  msec  (the  service 
time  of  a  routing  update  packet).  With  no  other  packets  in 
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the  systeii^  the  routing  update  packet  is  ieeediately  served 
at  a  rate  of  one  second  per  seconds  and  exits  the  channel 
after  23.2  msec.  After  the  departure  there  is  nr  work  in 
the  system  until  the  next  arrival. 

A  more  interesting  measure  of  performance  is  wlsn)^  the 
waiting  time  for  the  nth  message^  where  the  constant  in¬ 
terarrival  time  of  the  stream  traffic  is  s.  Figure  6.7  is  a 
plot  of  U(sn)  for  s  =  ?48r  165^  and  126  msec  (i.e.#  the 
periods  used  in  the  measurement  experiments).  For  these 
values  of  s  (and  the  message  size  used  in  the  measurement 
experiments)  the  line  utilization  is  > n  thr  20  to  40%  range 
so  that  every  fifth  update  is  dropped.  These  single  channel 
curves  nicely  display  some  of  the  characteristics  of  delay 
shown  in  Figures  4.1  through  4.4.  The  major  shape  of  Figure 
4.1  is  nearly  the  same  as  U(124n).  U(165n)  ana  U(24Pr) 
display  the  relative  variation  of  the  previous  correspondi no 
figures.  That  is  U(165n)  varies  more  rapidly  than  does 
Li(24Sn)  which  in  turn  varies  more  rapidly  than  U(124n);  and 
correspondingly  Figure  4.4  varies  more  rapidly  then  do  Fig¬ 
ures  4.2  and  4.3  which  in  turn  vary  more  rapidly  than  Figure 
4.1.  One  can  clearly  identify  the  period  of  each  U(sn) 
curve.  Indeed^  let  P(s)  represent  the  period  of  U(sn). 
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Figura  4.7.  Unfinithtd  work  at  ttroam  arrivals. 


Then 


P(s)  =  LCM(s/r) /s 

where  LCMCx^y)  =  the  least  common  multiple  of  x  and  yr  and  r 
=  the  routing  period  (i.e.^  the  minimum  time  r  such  that  the 
pattern  of  routing  arrivals  is  the  same  in  the  intervals 
(C^r)ir  (r#2r)»  <2r/3r)#  ...).  For  these  examples^  r=6Ad 
msec.  P(s)  has  the  values  ‘10#  123  -  160  for  s  ~  2AP/  ^  65  f 
and  12A  respectively. 

Figure  A. 2  shows  the  detailed  precession  of  the  periods 
of  the  routing  and  stream  traffic.  Each  row  in  Figure  A  .3 
shows  one  period  of  the  stream  data.  The  relative  positions 
of  the  data  (S)  packets  and  the  routing  (R)  packets  within 
the  period  are  shown.  Notice  that  every  fifth  routing  up¬ 
date  packet  is  missing.  The  figure  makes  clear  the  fact 
that  the  priority  which  routing  update  packets  carry  has  no 
effect  (in  the  absence  of  ambient  data  traffic).  That  i s^ 
data  packets  are  delayed  (by  routing  update  packets)  only 
when  they  arrive  while  the  channel  is  busy  serving  a  routino 
update  packet.  Since  preemption  is  not  permitted  on  the 
channels#  occasionally  a  routing  update  packet  must  wait  for 
a  data  packet  to  complete  service  on  the  channel  before 
being  transmitted. 
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Figure  4.8<ci.  PreoMion  of  intfferenco  potttm  (t*124). 
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«<hen  packets  pass  through  more  than  one  tandeir  channels 
a  more  complicated  pattern  arises.  An  example  pattern  for  a 
system  of  two  tandem  channels  is  shown  in  Figure  A. <5.  Once 
the  packets  are  delayed  at  channel  their  arrival  rate  at 
channel  Z'  then  corresponds  to  that  of  the  routing  update 
packets.  Durino  this  time  we  have  two  deterministic  streams 
arriving  at  a  constant  offset.  Hence  the  waiting  time  at 
ctiannel  Z  remains  constant  (except  for  routina  drop  auts) 
until  the  data  packets  are  no  longer  delayed  at  channel  1. 
This  example  illustrates  the  c I i mbi n g -d r opp i ng  phenomenon 
shown  in  Figures  4.2  and  4.3. 

An  exact  solution  of  the  n-node  tandem  server  model  is 
highly  dependent  on  the  phasing  of  the  routing  updates  on 
the  various  channels^  as  seen  in  Figure  4.9.  This  dependen¬ 
cy  produces  an  unwieldy  solution.  Therefore/  in  the  next 
section/  we  use  simulation  to  study  a  model  which  is  more 
closely  tied  to  the  actual  system. 

4.4  SimuifiiifiD 

Having  shown  the  exact  behavior  for  the  single  channel 
system/  we  now  resort  to  simulation  to  illustrate  some 
phenomena  present  in  the  more  complicated  tandem  system. 
Essentially  identical  experiments/  as  described  in  the  above 
measurement  section/  were  performed  usina  a  rather  detailed 
simulation  of  the  ARPANET.  The  simulation  program  was  writ¬ 
ten  in  PL/I  for  the  IBK  360/91  at  UCLA.  There  were  three 
mein  differences  between  the  measurement  and  simulation.  In 
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the  simulation^  the  ambient  data  traffic  was  set  to  7 » r o / 
node-to-node  acknowledgements  were  not  used/  and  the  fi^asiny 
of  the  routing  update  packets  on  the  va'-ious  channels  u-js 
rot  synchronized  with  that  of  the  measurement  experiments. 
Therefore/  we  expect  to  see  the  same  period  displayed  but 
possibly  a  different  shape  for  the  delay  curves.  Indee'^ 
Figures  4.1  and  4. 10  show  a  curve  with  a  period  of  32f  mes¬ 
sages.  Notice  that  the  period  differs  from  that  of  Ufsn) 
for  this  data  rate.  This  is  due  to  *'he  fact  that  while  the 
data  rate  causes  a  stable  rate  of  fn^r  routing  upoates  per 
basic  period/  tne  return  path  carrying  R^ny's  alternates 
between  four  and  five  updates  per  basic  period.  This 
results  in  a  stable  rate  of  nine  updates  in  each  two  basic 
periods.  That  iS/  12^0  is  the  minimum  time  r  for  which  the 
pattern  of  routing  arrivals  is  fixed  in  the  intervals  (O/r)/ 
(r/2r)/  <2r/3r)/...  .  Therefore/  the  round-trip  delay  curve 
has  a  period  of  L  CM  <  1  24/ 12*10)  /  1  24  =  320  messages.  Ficure 
4.1  may  exhibit  a  period  of  160  messages  due  to  the  node- 
to-node  acknowledgements  which  elevate  the  traffic  on  the 
backward  channel  just  enough  to  force  a  constant  rate  of 
four  updates  per  basic  period.  One  also  notices  that  the 
minimum  values  for  Figures  4.1  and  4.10  are  not  the  same. 
This  is  due  to  several  factors.  In  the  simulation  we  have 
estimated  the  channel  propagation  time  and  the  nodal  pro¬ 
cessing  time;  both  are  likely  lower  than  their  actual  value. 
Another  factor  is  that  we  have  assumed  zero  acceptance  time 
for  the  message  at  the  destination  node  and  for  the  PFNM  at 
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the  source  node.  These  assumptions  drive  the  delay  down  for 
the  simulation  and  hence  lowers  the  vertical  offset- 

Figures  A.?  and  4.3  are  too  random  to  identify  a 
period.  however^  one  should  notice  the  slow  climbirq  and 
rapid  falling  in  the  curves  in  Figures  4.2^  4.3#  4.11  and 

4.12  (though  the  measured  data  is  quite  noisy).  The  most 
rapid  variation#  both  for  simulation  and  measurement#  is  the 
packet  rate  of  approximately  six  per  second  shown  in  Figures 

4.13  and  4.4  respectively.  Roth  curves  possess  a  period  of 
128  messages . 

i<cstled  between  the  large  delays  in  Figures  4.1  and 
4.10  are  some  small  mounds.  These  are  due  to  th<*  interfer¬ 
ence  between  routing  updates  and  stream  traffic  in  the 
"TASK"  queue  [h'cdu  72D  (i.e,#  data  packets  waiting  to  be 
placed  on  an  output  queue  and  routing  update  oackets  weitino 
and  being  digested  into  the  local  routing  table).  The  TASK 
queue  is  currently  served  in  FCFS  fashion.  A  conceptually 
simple  modification  is  to  serve  routing  update  packets  at 
low  priority  from  the  TASK  queue.  (This  is  discussed  at 
length  in  Chanter  3.)  Ficures  4.14  through  4.17  show  the 
effect  of  sending  routing  updates  at  the  same  rate  as  be¬ 
fore#  but  processing  data  packets  by  preempting  the  process¬ 
ing  of  interfering  routing  updates.  Notice  the  decrease  in 
both  the  average  and  variance  of  delay.  These  curves  show 
the  best  possible  delay  under  the  current  periodic  update 
scheme#  assuming  preemption  is  not  allowed  on  the  channels 
as  well. 
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Figure  4.12.  Round-trip  delay  simulation  (5  hops,  s=248|. 
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Figure  4.13.  Round-trip  delay  simulation  (2  hops,  s«165). 
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Figure  4.16.  Round-trip  delay  simulation  with  low  priority  routing  processing  (5  hops,  s=248}. 
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Figur  '  4.17.  Round-trip  delay  simulation  with  low  priority  routing  processing  (2  hops,  S’^ldS). 


for  sending  strean  traffic  in  an  otherwise  eapty  net¬ 
work#  the  processing  of  routing  updates  at  low  priority  pro¬ 
vides  superior  performance  to  the  current  FCFS  approach. 
There  may  be  some  question  whether  this  superiority  would 
reaain  in  a  loaded  network.  As  the  traffic  increases#  the 
routing  updates  remain  on  the  TASK  queue  for  a  longer  time 
tnus  causing  a  possible  lag  in  the  propagation  of  recent 
routing  information.  Eventually#  one  might  expect  that  lag 
would  result  in  worse  routes  being  used#  and  correspondingly 
increased  delays.  This  possibility  must  be  compared  against 
the  added  delay  to  data  traffic  in  the  FCFS  case.  Analysis 
of  a  single  node  in  isolation  as  well  as  the  results  of  a 
simulation  of  a  network  appear  in  Chapter  3#  with  the  con¬ 
clusion  that  routing  update  processing  at  low  priority  is 
gooa  Doth  locally  and  globally. 

Cne  may  go  one  step  further  and  consider  the  perform¬ 
ance  of  a  system  where  routing  update  packets  may  be 
preempted  from  service  on  the  channels  as  well.  This  "total 
background"  routing  may  be  difficult  to  implement  in  prac¬ 
tice.  The  next  step#  if  we  ignore  network  component 
failures  for  the  moment#  is  to  eliminate  routing  updates  and 
to  use  some  form  of  fixed  routing.  As  with  total  background 
routinn#  this  causes  no  interference  of  data  packets  due  to 
routing  information  processing.  One  wonders  if  fixed  rout¬ 
ing  will  be  less  effective  since  it  cannot  adapt  to  changing 
traffic  patterns  (i.e.#  at  very  low  traffic  levels  fixed 
routing  performs  better  than  adaptive  schemes#  but  perhaps 


may  fail  at  higher  traffic  levels).  Homever^  ye  do  recall 
from  CKlei  643  that  properly  designed  fixed  routing  pro¬ 
cedures  may  be  superior  to  adaptive  ones.  This  is  further 
supported  by  LPric  723.  Figure  4.18  also  shows  the  relative 
performance  of  a  fixed  scheme  (FR>  and  the  two  others  (HOR 
and  LLFLR  as  described  in  Chapters  2  and  3).  To  give  per¬ 
spective  here^  we  note  that  a  relative  traffic  load  of  1.0 
is  slightly  higher  than  the  weekly  average  traffic  level 
reported  in  tKlei  743.  We  note  also  that  with  a  high  degree 
of  confidence^  the  values  for  mean  delay  are  correct  to 
within  ±7%  for  the  first  three  levels  of  relative  traffic 
intensity  (i.e.#  1.0^  5. Or  and  10.0).  The  values  above 
intensity  10.0  are  less  precise#  but  the  only  auesticnable 
points  are  for  LL^LR  and  FR  at  load  14.  The  range  of  values 
for  LLFLR  and  FR  overlapped  in  the  several  simulations  for 
this  intensity.  The  fixed  routes  used  were  the  shortest  hop 
paths.  One  could  choose  an  even  better  fixed  routing  scheme 
by  using  the  flow  deviation  method  CGerl  733  for  example. 
Notice  that  fixed  routing  always  performed  better  than  the 
foreground  routing  (HOR)#  and  only  at  very  high  traffic  lev¬ 
els  is  it  worse  than  the  background  processing  routing 
(LLFLR).  This  suggests  that  the  cost  of  routing  in  the 
ARPANET  is  extremely  high  indeed#  since  traffic  levels  are 
currently  very  low. 
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Figure  4.18.  Adaptive  routing  vs.  fixed  routing  (simulation). 


110 


Ue  have  so  far  ignored  network  failures.  It  is  clear 
that  failures  do  occur  in  practice.  Long  term  monitoring  of 
the  ARPANET  LMcKe  74]  shows  a  mean  time  between  failures 
(MTBF)  of  431  hours  for  lines  and  221  hours  for  nodes. 
Failures  cause  topological  changes  to  occur  in  the  network 
in  the  following  two  ways.  When  a  channel  fails/  it  is  as 
if  it  were  removed  from  the  network.  when  a  node  fails/  all 
its  attached  channels  are  removed  from  the  network.  We 
define  the  network-wide  MTBF  to  be  the  mean  time  between 
channel  removals.  Then  with  the  *»7  nodes  and  65  full  duplex 
channels/  in  the  June  1975  ARPANET/  and  assuming  that  each 
node  is  of  average  connectivity  (i.e./  approximately  2.24 
CBBN  75])/  these  figures  yield  a  network-wide  KTBF  of  3.76 
hours . 

A  far  better  method  of  routing/  it  appears/  would  be  to 
use  a  "passive”  scheme.  In  such  a  scheme  one  establishes 
routes  and  continues  to  use  them  in  a  fixed  routing  fashion 
until  some  catastrophe  occurs  (i.e./  a  failure  or  possibly 
even  severe  congestion).  At  the  time  a  catastrophe  occurs 
one  could  "turn  on"  routing  updates  until  the  tables  "sta¬ 
bilize".  From  the  above  data  one  can  see  that  the  average 
time  oetween  turning  on  routing  due  to  failure  would  be 
almost  two  hours  (or  approximately  1D000  times  the  basic 
rout ing  period) ! 

Recently  a  technique  called  "explicit  path  routing"  was 
introduced  by  Jueneman  and  Kerr  CJuen  76].  This  scheme  pro¬ 
vides  for  a  set  of  fixed  paths  between  any  node  pair  and 
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requires  no  updating  of  routing  tables.  This  procedure 
deserves  further  Investigation  In  light  of  our  results. 

4.S  CQQ£iusiQQS 

In  this  chapter  we  have  shown  some  Interesting  message 
delay  phenomena  for  periodic  stream  traffic  attributable  to 
the  periodic  routing  scheme  used  In  the  ARPANET.  ke  con¬ 
clude  that  periodic  routing  Is  quite  costly  in  medium  sizeH 
(and  bigger)  networks.  In  order  to  assure  good  oerformance 
In  the  face  of  failure  (or  heavy  congestion)  one  pays  ?  high 
price  In  terms  of  message  delay  for  periodic  routing  update 
procedures  in  networks  of  the  size  of  the  ARPANET.  we  sug¬ 
gest  the  use  of  a  passive  routing  scheme^  In  which  updates 
are  scheduled  (only)  as  the  result  of  failure  (or  heavy 
congestion).  The  results  presented  here  suggest  that  this 
method  could  provide  superior  performance  at  reduced  cost. 
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CHAPTER  S 


(SOURCE  AND)  DESTINATION  BUFFERING  CONSIDERATIONS 
FOR  STREAM  TRAFFIC  COMMUNICATION 


5-1  lQtrody£^i2Q 

In  coRimuni  eating  stream  information  via  a  packet 
switched  network  rather  than  the  traditional  circuit 
switched  (or  dedicated)  network  there  arise  some  unique 
problems  which  require  solution.  Among  these  problems  is 
the  packaging  of  information  into  packets.  Some  ad  hoc 
solutions  to  this  problem  are  discussed  in  section  5.2.  ?ut 
the  chief  concern  of  this  chapter  is  dealing  with  the  vari¬ 
able  delay  imposed  by  a  packet  switched  network.  Several 
methods  of  limiting  this  variability  in  the  output  by  desti¬ 
nation  buffering  are  considered-  3y  delaying  the  output  of 
the  first  message  of  a  stream^  one  may  limit  the  frequency 
and  duration  of  gaps  in  the  output  (i.e.^  intervals  0+  time 
in  which  no  data  is  available  to  output).  Such  gaps  are 
undesirable  since#  to  some  extent#  they  destroy  the  rhythm 
of  the  output  and  thus  hinder  the  intelligibility  of  the 
information.  Clearly#  as  the  first  message  delay  is  in¬ 
creased#  the  frequency  and  duration  of  gaps  decreases.  In 
the  limiting  case  of  infinite  destination  buffering  delay  it 
is  guaranteed  that  zero  gaps  will  occur  in  the  output  but 
this  of  course  destroys  the  interactive  nature  of  the 
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coaaunication.  Therein  lies  the  tradeoff  which  is  examined 
in  this  chapter^  namely  gaps  versus  delay. 

tiy  assuming  statistical  independence  of  the  delay  ex¬ 
perienced  by  stream  packets  traversing  the  networks  we  may 
analyze  this  tradeoff.  The  assumption  is  somewhat  justi¬ 
fied^  but  fails  to  be  true  under  certain  conditions^  which 
we  partially  examine.  Based  on  this  independence  assumption 
we  can  derive  some  general  analytic  results.  Numerical 
results  are  presented  for  the  exponential  distribution  in 
some  detail  and  to  a  lesser  extent  for  a  class  of  r-stage 
Erlangian  distribution  of  delay.  Following  the  theoretical 
performance  results/  we  present  some  results  of  a  simulation 
which  allows  for  the  relaxation  of  some  of  the  assumptions 
of  the  model.  This  is  accomplished  with  the  use  of  a  trace 
driven  simulation  of  the  various  buffering  methods. 

5.2  iSQsiiQS  Itfaiegy 

whenever  stream  information  is  produced  by  the  source 
in  units  which  are  smaller  than  a  full  packet  (or  message) 
we  have  the  option  of  sending  the  information  in  partially 
full  packets.  Doing  so  reduces  the  time  required  to  create 
a  packet/  but  increases  the  throughput  requirement  of  the 
source  since  packets  contain  a  non-zero  amount  of  overhead. 
:>nce  again/  one  finds  a  tradeoff  between  throughput  and 
delay! 

This  tradeoff  is  pictured  in  Figure  5.1  for  the  fixed 
rate  ARPANET  LPC  algorithm  (see  CCohe  76D).  There  is  a 
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linear  increase  in  packet  fill  time  as  we  move  from  the 
fflinfmutn  of  one  parcel  to  the  maxieuiR  of  14  parcels  per  pack¬ 
et.  (A  parcel  consisting  of  67  bits  contains  one  set  of  LPC 
coefficients  and  as  such  is  the  smallest  unit  which  iray  be 
separately  interpreted  at  the  destination.)  The  slope  is 
19.2  msec/parcel.  It  is  clear  that  an  inverse  relationship 
holds  for  the  throughput  in  terms  of  either  oackets/sec  or 
bits/sec  as  shown.  If  the  network  protocol  uses  a  full 
packet  time  to  send  a  partially  full  packet  (e.g.»  a  slotted 
channel  as  in  the  ARP A  SAT NET  iKlei  73D  and  PRNET 
CKahn  75])^  then  the  bits/sec  curve  is  of  no  use. 

Two  extreme  aporoaches  exist  for  a  sending  strategy: 
(1)  minimize  delay  (at  whatever  the  resulting  throughput 
requirement);  and  (2)  minimize  the  throughput  requirement 
(at  whatever  the  resulting  delay).  It  is  clear  that  the 
choice  of  sending  full  oackets  satisfies  (2).  Powever^  it 
is  not  so  clear  that  sending  the  smallest  packets  minimizes 
the  overall  delay. 

Let  f(p)#  x(p)^  c<p)/  w(p)/^  and  P(p)  be  defined  as  fol¬ 
lows  for  the  stream  packets  given  that  packets  of  size  o 
parcels  are  in  use. 

f  (p)  =  the  fill  time# 

x(p>  =  the  network  transmission  time# 

c(p)  =  the  network  propagation  and  processino  time# 

w(p)  =  the  average  network  waiting  time#  and 

D(p)  =  the  average  destination  buffering  time. 
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{ aat/mq 


We  wish  to  nininize 


ftp)  +  x(p)  +  c(p)  ♦  w(p)  Dtp)  tS.1) 

over  the  choice  of  p  between  1  and  14.  While  ftp)  and  xtp) 
increase  with  p/  wtp)  and  Dtp)  decrease  as  p  increases. 
Since  channel  utilization  decreases  as  p  increases  we  expect 
the  average  waiting  time  to  decrease.  This  in  turn  causes  a 
decrease  in  the  variation  of  delay  and  therefore  htp)  may  be 
decreased  to  give  the  same  gao  probability  (see  section 
5.3).  Also#  for  the  same  (packet)  gap  probability#  as  the 
number  of  packets  increases  (as  p  decreases)  more  parcels 
are  effected  by  gaps#  and  tnus  b  must  be  increased  tc  pro¬ 
vide  equal  intelligibility.  Therefore  is  is  difficult  to 
find  that  p  which  minimizes  overall  delay. 

Between  these  two  extreme  sending  policies  there  are 
many  alternative  approaches  (e.o.#  (a)  minimize  delay  at 
"reasonable"  throughput  requirement).  Beyond  p=5  the 
throughput  curves  are  relatively  flat  (i.e.#  the  throughput 
requirement  does  not  change  significantly  between  p=5  and 
p=14).  Therefore  a  choice  of  p=5#  6#  or  7  seems  to  fit 
approach  (a) . 

ea£!i£i  &iz£ 

One  would  like  to  go  a  step  further  in  minimizing  eoua- 
tion  5.1#  and  use  packets  of  variable  size.  For  example# 
one  may  wish  to  use  smaller  packets  when  w  is  likely  to  be 
large  thus  reducing  f  and  x#  and  conversely  seno  larger 
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packets  when  w  is  likely  to  be  small  enough  to  guarantee 
that  overall  delay  will  not  exceed  a  tolerable  level. 

An  examination  of  this  approach  has  shown  that  on  a 
microscopic  scale  (i.e./  packet  by  packet)  it  is  infeasible 
since  it  was  found  that  future  delay  is  not  well  predicted 
by  past  delay  in  general.  The  correlation  results  presented 
in  section  5.3.3.i!  attest  to  this.  Therefore  durino  the 
short  term  it  is  wise  to  use  fixed  p. 

On  a  macroscopic  scale  some  adjustment  of  p  may  prove 
useful.  This  may  be  accomplished  by  us’no  the  destination 
monitoring  techniques  discussed  in  section  5.3.  As  network 
delay  increases  one  may  wish  to  use  smaller  p.  This  may  be 
done  only  when  w  is  not  significantly  increased  by  this 
decrease  in  p.  If  network  delay  increases  quite  substan¬ 
tially  then  an  increase  in  p  is  in  order  (to  reduce  w). 

•e  have  offered  here  only  ad  hoc  suggestions  for  the 
sending  process.  With  a  good  model  for  w(p)  some  useful 
analytic  results  would  be  attainable.  But  since^  as  pointed 
out  earlier/  the  throughput  requirement  curves  are  fairly 
flat  oast  p=5/  the  payoff  gained  by  such  analysis  is  likely 
to  be  minimal.  We  shall  therefore  direct  our  attention  to 
the  receiving  process, 

5.3  A^gciiyg  receiving 

The  aim  of  adaptive  receiving  of  stream  traffic  is  to 
output  the  information  with  as  close  to  the  same  timing  with 
which  it  was  originally  generated  under  the  constraint  of 
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allowing  interactive  communication  (i.e.»  reasonably  smalt 
delay).  It  is  particularly  important  in  the  case  of  speech 
to  reduce  the  frequency  and  duration  of  qaps  in  order  to 
insure  acceptable  intelligibility.  In  this  section  we  first 
describe  three  methods  of  gap  control  using  buffering^  and 
two  playback  strategies  for  handling  those  gaps  which  slip 
through  the  control  procedures-  The  performance  of  the 
various  schemes  is  then  analyzed.  We  then  present  some 
numerical  results  for  the  theoretical  performance^  followed 
by  simulation  results  comparing  the  various  proceciures. 

5.3.1  Gag  cgn i. rg I 

Since  variable  delays  do  occur  in  a  packet  switched 
network/  if  no  smoothing  were  done/  gaps  would  occur  ir  the 
output  of  a  packet  stream.  That  is/  there  would  be  periods 
of  time  in  which  no  data  was  available  Tor  playout.  T^'is 
has  disastrous  effects  on  the  underst andabi I ity  oT  speech  in 
particular  and  in  general  violates  the  definition  of  stream 
traffic.  Therefore  gap  prevention  or  reduction  is  neces¬ 
sary. 

One  method  of  gap  control  is  to  delay  the  output  of  the 
beginning  of  a  sentence  (i.e./  a  period  of  activity  by  the 
sender)  by  an  amount  0/  which  may  be  selected  at  the  begin¬ 
ning  of  each  sentence  based  on  samplino  of  previous  delays. 
Another  scheme  is  to  slow  the  playout  in  the  absence  of  the 
next  packet  and  speed  playout  when  an  excess  of  weitinq 
packets  exists.  (The  effect  oT  tnis  on  intelligibility  may 
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not  warrant  its  use.)  One  «ay  ■ix  these  two  to  arrive  at 
still  others  or  invent  new  ones.  We  shall  consider  only  the 
first  scheme. 

The  first  packet  of  a  sentence  will  be  delayed  by  an 
amount  called  the  destination  wait  time  denoted  as  D.  The 
choice  of  D  should  be  made  large  enough  to  reduce  the  fre¬ 
quency  of  gaps  to  a  tolerable  level  but  small  enough  to 
retain  the  interactive  nature  of  a  conversation.  In  a  later 
section  we  shall  consider  several  adaptive  methods  for 
choosing  D. 

5.3.2  PlgybiSk  mgiliQiis 

Unless  0  is  very  large/  there  is  a  non-zero  probability 
that  a  gap  will  occur.  The  purpose  of  this  section  is  to 
describe  two  methods  of  dealing  with  this  eventuality.  The 
first  method  (method  E)  would  expand  the  playout  time  of  a 
sentence  in  order  to  include  all  packets  in  the  output  pro¬ 
cess.  The  second  method  (method  I)  preserves  the  timing  at 
the  expense  of  ionoring  some  late  arriving  packets  (or  par¬ 
tial  packets).  These  two  approaches  lie  at  opposite  ends  of 
o  continuum  of  choices  for  dealing  with  gaps.  ^t  constant 
delay  (e.g./  dedicated  channel)  the  two  extremes  are 
equivalent.  Also  if  0  is  infinite  the  extremes  coincide, 
however/  with  finite  0  and  variable  delay  the  extremes 
separate.  The  separation  increases  as  delay  variability 
increases  or  as  0  decreases.  At  infinite  delay  variability 
method  E  requires  infinite  time  to  output  a  sentence  (of 
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more  than  one  packet)/  while  method  I  would  output  only  the 
first  packet  of  a  sentence  and  ignore  any  other  packets. 
Thus  method  E  preserves  the  information  at  the  expense  of 
the  interpacket  timing/  and  method  I  preserves  the  intei — 
packet  timing  at  the  expense  of  discarding  some  information, 
with  finite  delay  variability  and  finite  D  one  can  envision 
methods  which  lie  between  the  two  extremes.  For  example/ 
one  may  wish  to  discard  only  those  packets  which  arrive  both 
late  and  out  of  order  <i.e./  a  packet  is  discarded  if  its 
successor  is  currently  being  output).  Such  a  scheme  has 
properties  of  both  methods  E  and  I.  The  time  axis  is  ex¬ 
panded  when  a  packet  is  just  late/  yet  some  data  may  be  dis¬ 
carded  in  order  to  preserve  “reasonable”  timing.  Another 
example  is  to  limit  the  expansion  of  time  and/or  the  frac¬ 
tion  of  discarded  data  to  a  certain  amount  and  switch 
methods  if  the  threshold  is  exceeded. 

An  important  consideration/  which  shall  not  te  dis¬ 
cussed  here/  is  the  filling  of  gaps  (i.e./  with  silence  or 
something  else).  Several  alternatives  (which  are  beyond  the 
scope  of  this  discussion)  have  been  used  and  are  discussed 
in  CForg  761. 


5.3.2. 1  LacaDdfid  tiss  axis  f55£lb2d 
In  this  method  content  is  regarded  as  the  most  impor¬ 
tant  part  of  the  stream  of  packets.  Therefore/  it  is 
desired  to  deliver  all  packets  which  are  produced  by  the 
source  of  the  stream  traffic.  Timing  is  considered 
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iaportant  only  to  the  extent  that  a  packet  is  delivered  in 
the  sane  sequence  in  which  it  was  generated.  A  "late"  pack¬ 
et  is  one  which  arrives  after  it  is  desired  for  cutout 
(i.e./  after  its  predecessor  has  finished  olayout).  In  this 
method  a  late  packet  causes  all  succeeding  packets  to  be 
delayed  in  playout#  thus  expanding  the  time  axis  or  the 
length  (in  time)  of  the  sentence.  An  example  of  this  is 
illustrated  in  Figure  5.2(a). 

This  is  the  general  approach  taken  by  the  network 
speech  compression  research  group  at  the  University  of 
Southern  California  Information  Sciences  Institute  (ISI). 
In  the  ISI  scheme  extremely  late  out  of  order  packets  are 
i gnored. 


5. 3. 3. 2  Lais  dais  ianacs-st  i> 

In  method  I  timing  is  considered  to  be  of  primary  im¬ 
portance.  A  packet  (or  partial  packet)  will  be  used  only 
when  it  is  not  late.  By  discarding  late  information  it  is 
always  possible  to  retain  the  sentence  length  (in  time). 
Figure  5.2(b)  shows  the  previous  example  sentence  from  Fig¬ 
ure  5.2(a)  passing  through  method  I.  Notice  that  sentence 
time  is  preserved  but  that  more  gaps  occur  than  in  method  f. 
This  is  true  in  general  as  we  shall  see  in  later  sections. 
Method  I  is  in  use  by  experimenters  at  the  Massachusetts 
Institute  of  Technology  Lincoln  Laboratory. 
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Figure  5.2.  Ptayout  Methods. 
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5.3.3  EfittficiaQ&fi  fi^aiuitiaD 

An  examination  of  the  performance  of  several  adaptive 
receiving  techniques  is  presented  in  this  section.  Ue  begin 
by  considering  some  assumptions  which  render  the  system 
tractable  for  analysis.  The  analysis  is  then  performed.  An 
attempt  to  characterize  communication  quality  is  made. 
Numerical  results  based  on  the  analysis  are  presented  in 
detail  for  the  exponential  distribution#  and  in  lesser  de¬ 
tail  for  a  class  of  t^rlangian  distributions  TKlei  75D. 
Simulation  is  used  to  show  some  results  with  the  assumptions 
relaxed. 

Our  model  of  the  system  is  pictured  in  Figure  5.3.  a 
period  of  activity#  called  a  sentence#  is  initiated  by  the 
sender  at  some  time.  (This  corresponds  to  the  detection  of 
no-silence  in  speech  for  example.)  As  time  prooresses  the 
sentence  is  broken  up  into  a  sequence  of  segments  called 
packets  or  messages.  A  finite  amount  of  time  f(i)  is  re¬ 
quired  to  fill  packet  i.  Also  associated  with  each  packet  1 
is  a  network  transit  delay  y(i).  Let  t(i)  =  f(i)  +  y(i>  be 
the  source-to-dest ination  delay  of  packet  i.  We  make  the 
following  assumptions  regarding  the  random  variable  t(i): 
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communication. 


AzsUIBlifiDS  cl  ttl£  Ofidfii 

1.  Independence 

We  shall  assume  that  each  t(i)  is  chosen  indepen¬ 
dently  from  a  probability  distribution  function 
S(2)  =  Pr[t<i)<z3. 

2.  Stationarity 

We  shall  further  assume  that  Slz)  is  stationary 
(i.e.#  S(z)  is  not  a  function  of  time). 

While  the  assumption  of  stationarity  is  used  in  order 
to  obtain  analytic  results^  the  algorithms  (as  we  shall  see) 
adapt  to  nonstationary  behavior. 

Following  the  source-to-dest i nat i on  delay  there  is  a 
(possibly  zero)  destination  buffering  delay  for  each  packet. 
In  particular^  this  buffering  delay  takes  the  value  I  for 
the  first  packet  of  a  sentence.  As  we  see  later/  D  is  a 
function  of  S(z)  and  the  particular  delay  monitoring  algo¬ 
rithm  in  use. 

It  is  assumed  that  each  packet  requires  f(i)  to  empty 
(as  well  as  to  fill).  If  a  packet  arrives  after  its  prede¬ 
cessor  has  completed  emptying/  then  a  gap  occurs  in  the  out¬ 
put  stream  (indicated  in  the  figure  by  crosshatching  at  the 
output).  Obviously  there  is  a  tradeoff  between  the  value  D 
and  the  frequency  with  which  gaps  occur.  We  shall  therefore 
use  these  two  parameters  as  our  measures  of  performance. 
Let  us  first  consider  the  validity  and  implication  of  the 
major  assumptions  of  the  model. 
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s.3.3.1  fil  QfitMQCk  dfiiay 

The  purpose  of  this  section  is  to  present  some  observed 
distributions  of  network  delays  and  provide  guidelines  for 
producing  tractable  approximations  of  these  distributions. 
Both  ARPANET  measurement  and  simulation  were  used  obtain 
delay  distribution  examples. 

The  approximation  derived  from  the  independence  assump¬ 
tion  of  Kleinrock  CKlei  64D  would  suggest  an  Erlangian 
(i.e.#  the  sum  of  exponentials)  distribution  of  delay.  This 
assumption  renders  the  model  of  each  channel  an  queue 
which  yields  an  exponentially  distributed  system  time.  The 
time  through  a  series  of  queues  therefore  would  follow  an 
Erlangian  distribution. 

exact  formulas  have  teen  derived  for  the  distribution 
of  message  delay  in  an  isolated  non-interf ered  path  in  a 
network/  for  Poisson  arrivals  and  deterministic  service 
time/  by  Rubin  (see  CRubi  75D).  Our  problem  is  to  find  the 
delay  distribution  of  a  stream  traffic  source  in  a  general 
network  which  does  not  seem  to  fit  the  assumptions  of  the 
previous  two  models.  Namely  the  stream  oackets  must  not  be 
allowed  to  change  length  as  they  proceed  through  the  net¬ 
work/  and  our  problem  would  be  trivial  in  a  non-interf ered 
path . 


5. 3. 3. 1.1  BeaaucsffiftQt 

The  measurement  experiments  described  in  Chapter  4  pro¬ 
vide  the  distributions  presented  here.  In  order  to 


ellninate  the  start-up  effects  the  first  100  sanples  have 
been  discarded  in  each  case.  The  resulting  histograms  are 
plotted  in  Figure  5.A(a-d).  In  (a)  and  (b)  there  is  a  tri- 
modal  behavior  exhibited.  This  shows  the  effect  of  the 
periodic  routing  update  procedure  quite  clearly  (again). 
The  first  mode  corresponds  to  the  round-trip  delay  experi¬ 
enced  by  a  message  which  always  finds  empty  queues  along  its 
path.  The  third  mode/  falling  about  45  msec  after  the  first 
mode/  corresponds  to  the  delay  of  those  messages  and 
corresponding  end-to-end  acknowledgenents  (RFNM)  which  must 
wait  for  a  total  of  45  msec  on  queues  while  traversing  the 
network.  This  amount  of  time  corresponds  to  the  transmis¬ 
sion  time  of  two  routing  update  packets/  the  processing  of 
four  such  packets  (approximately  two  in  the  slower  316 
nodes)/  or  some  combination  of  the  above.  The  second  mode/ 
occurring  roughly  10  msec  beyond  the  first/  is  probably  due 
to  the  packet  (or  its  RFNM)  waiting  behind  the  processing  of 
one  routing  update  along  the  way.  Significance  could  prob¬ 
ably  be  assigned  to  the  other  peaks  as  well  but  this  would 
belabor  the  point. 

As  we  move  to  five  hops  (c)  and  then  to  ten  hops  (d)  we 
notice  that  the  trimodal  behavior  ceases  to  appear  and  the 
histograms  take  the  shape  of  an  Erlangian  density.  Perhaps 
Kleinrock's  independence  assumption  produces  an  acceptable 
approximation  for  stream  traffic  as  well! 
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5. 3. 3. 1.2  iiouiallQD 

The  results  of  the  measurement  are  biased  by  the  large 
overhead  associated  with  the  routing  update  procedure.  We 
therefore  resort  to  simulation  in  order  to  remove  this  ef¬ 
fects  to  control  the  level  of  interfering  traff-’c  in  the 
networks  and  to  study  one  way  delay  instead  of  round-trip 
delay.  The  simulation  had  the  following  characteristics: 

Topology:  A  ring  of  21  nodes  and  21  full  duplex  chan¬ 

nels 

Channel  capacity:  5C000  bits/sec 

Traffic  pattern:  A  uniform  traffic  matrix  of  exponen¬ 
tially  distributed  message  lengths  with  a  mean  of 
500  bits  and  Poisson  arrivals  as  background  traff¬ 
ic.  Inserted  with  this  was  one  stream  traffic 
source  sending  to  a  destination  ten  hops  away  at  a 
rate  of  one  500  bit  packet  every  25?  msec. 

The  background  traffic  was  set  to  three  part’cular  lev¬ 
els  in  order  to  produce  .5  and  .9  channel  utilization. 

One-way  network  delay  was  measured  for  the  stream  traffic 
and  the  resulting  histograms  apoear  in  Figure  5.A(e-g).  The 
.1  load  histogram  (e)  suggests  a  shifted  exponential  density 
with  perhaps  an  impulse  at  the  shift  value.  For  load  .5  and 
.9  If  and  g  respectively)  the  histograms  have  more  of  an 
Erlanaian  shape  (with  an  impulse  in  the  .5  case). 
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Figure  5.4(c|.  Delay  histogram  (measurement,  5  hops) 
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histogram  (measurement.  10  hops). 


5. 3. 3. 1.3  iciciabig  aBccaiiaaticQs 


In  order  to  obtain  some  analytic  results  we  must  assume 
that  the  delay  distribution  may  be  characteriEed  in  a 
mathematically  tractable  form.  In  this  section  we  consider 
some  tractable  approximations  to  the  measured  and  simulated 
delay  distributions.  We  have  already  suggested  two  -  (1) 

the  shifted  exponential  and  (2)  the  Erlangian  class, 
shifie^  aXBQQSQiial  (liSlribu^.i  on 
This  distribution  may  be  formulated  as 


/ 


0 


S(t) 


1  -(t-b)/w 

-  e 
w 


t<b 


t>b 


where  bis  the  amount  of  shift  and  b+w  is  the  mean  value. 
Figure  5.5  shows  the  histogram  of  the  simulation  in  figure 
5.4<e)  plotted  together  with  a  shifted  exponential  with  the 
same  mean  value  and  with  b  equal  to  the  minimum  observed 
value.  This  appears  to  be  a  fairly  close  tit. 

IhS  LciaQfl  fiEiiZ  si  distcibytiani 

The  shifted  exponential  was  a  close  fit  to  one  of  the 
sample  distributions.  Je  now  consider  the  Frlang  family 

t 

which  may  bt  formulated  as 


d5(t)  =  ■■■■■■'  dt 

(r-1) ! 
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May  (iroac) 


Figura  S.5.  Shiftad  axponantial  fit  to  drfay  hiitogram  (liimilation,  load  -  .1 ). 


The  foraula  has  been  written  so  that  mean  value  is  indepen¬ 


dent  of  r.  As  Kleinrock  CKlei  7S3  points  out  on  page  124^ 
the  standard  deviation  of  this  distribution  is  x/sqrt(r). 
(Figure  5.26  shows  this  density  function  for  several  values 
of  r . ) 

By  selecting  r  one  may  change  the  coefficient  of  varia¬ 
tion  (defined  to  be  the  standard  deviation  divided  by  the 
mean)  between  zero  and  one.  All  the  observed  distributions 
have  coefficients  of  variation  in  this  range.  Figure  5.6 
shows  those  observed  histograms  (which  appear  to  be  Erlangi- 
an)  plotted  together  with  the  member  of  the  Erlang  family  of 
the  appropriate  r  and  mean  value.  This  appears  to  be  a 
close  fit^  particularly  for  the  simulation. 
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Figurt  S.6(b).  Erlang  fit  to  daiay  histogram  (maasHramatrt,  10  hops). 
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Eilang  fit  to  delay  hiatogram  (timulation,  load  •>  .5). 
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Figure  5.6(d).  Erlang  fit  to  delay  histogram  (simulation,  load  >  .9). 
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Ibfi  shittfid  iciftQaiSD  disitibuticoi 

So  far  we  have  seen  that  neither  the  shifted  exponen¬ 


tial  nor  the  Erlang  family  is  by  itself  sufficiently  rich  as 
to  model  all  of  the  observed  behavior.  Since  each  class  did 
well  in  a  set  of  instances^  it  seems  natural  to  consider  the 
closure  of  the  two  classes.  The  resulting  class  -  the 
shifted  Erlangian  -  is  considered  here.  The  density  func¬ 
tion  for  this  class  may  be  written  as 


where/  as  before/  b  is  the  amount  of  the  shift  and  w  =  x 


b . 

Figure  5.7  shows  the  fit  of  the  histogram  of  the  ap¬ 
propriate  shifted  Erlanqian  distributions  together  with  the 
observed  histograms.  Notice  the  close  fit  for  all  but  the 
measured  distributions  for  one  and  two  hops.  The  two  which 
are  not  well  approximated  with  the  shifted  Erlanc  class 
could  it  appears  be  approximated  by  a  waited  sum  of  shifted 
exponentials/  but  we  shall  not  attempt  this  here. 
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Figure  5.7(b).  Shifted  Erlang  fit  to  delay  histogram  (measurement,  2  hops). 
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Figure  5.7(c).  Shifted  Erlang  fit  to  delay  hisogram  (measurement,  5  hops). 
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Figure  S.7(d).  Shifted  Erlang  fit  to  delay  histogram  (measurement,  10  hops). 
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5. 3. 3. 2  iodacfiDd£D£fi  cl  Qsia&ck  daiay 

In  the  study  of  stochastic  processes^  statistical  in¬ 
dependence  is  often  assumed  to  make  analysis  simpler  (even 
possible).  We  make  use  of  such  an  assumption  here.  In  this 
section  an  heuristic  argument  is  presented  to  justify  this 
assumption.  This  argument  is  then  partially  substantiated 
by  measurement  and  simulation. 

We  assume  that  one  of  the  characteristics  of  a  stream 
source  is  that  packets  are  emitted  at  relatively  large  in¬ 
tervals  compared  to  the  interval  during  which  a  packet  occu¬ 
pies  a  single  channel  in  the  network.  More  precisely  we 

have  the  property  that  the  average  interarrival  time  of 
packets  from  the  stream  source  t  >>  s  the  average  time  spent 
waiting  for  and  using  a  channel.  This  is  both  an  assumption 
of  (a)  moderately  low  throughput  for  the  stream  source#’  and 
<b)  low  overall  traffic.  If  (a)  were  not  true#  then  the 
source  becomes  classified  as  a  high  throughput  source.  If 
lb)  were  rot  the  case  then  the  network  delays  may  exceed 
those  desired  for  interactive  stream  communication. 

with  this  assumption  we  see  that  each  packet  enters  the 

network  and  is  likely  to  be  far  along  its  way  to  the  desti¬ 

nation  before  its  successor  enters  the  network.  Therefore 
each  packet  arrives  to  find  the  network  in  a  state  which  is 
"independent”  of  the  influence  of  his  predecessors  which  are 
no  longer  in  sight.  Each  packet  then  receives  an  indepen¬ 
dent  "look"/  if  you  will/  at  the  network.  We  conclude  that 
the  delay  experienced  by  successive  packets  should  be 


approximately  independent. 

Let  us  examine  the  measurement  and  simulation  results. 
A  necessary/  byi  naJ,  suffiaignt/  condition  for  statistical 
independence  is  that  of  linear  independence.  Linear  depen¬ 
dence  may  be  tested  simply  by  computing  the  correlation 
coefficient  CFell  57]  <p.  221)  of  a  sequence  of  delays  with 
itself  shifted  by  an  amount  j.  9y  changing  the  shift  j  we 
obtain  a  sequence  of  correlation  coefficients.  This  is 
related  to  the  "autocorrelation"  sequence  defined  in 
CCppe  75]  Cp.  384).  Cur  sequence  however  is  normalized  by 
first  subtracting  the  mean  from  each  value  (which  gives  the 
"autocovariance"  sequence)  and  dividing  the  result  by  the 
variance  of  the  original  sequence.  The  result  of  this  com¬ 
putation  for  3=1/2/  .../  200  is  shown  in  Figure  5.f(a-d) 
for  the  measured  sequences  of  delay  and  in  Figure  5.P(e-g) 
for  the  delay  sequences  produced  by  the  simulation  described 
in  the  previous  section.  The  figures  show  that  a  near  zero 
linear  dependence  exists  for  the  measured  delay  of  5  hops  or 
more  and  for  a  load  of  .1  in  the  simulation.  For  a  load  of 
.5  successive  delays  are  less  than  30  percent  correlated  but 
beyond  j=1  there  is  little  correlation.  As  expected  delay 
at  a  load  of  .9  is  highly  correlated  for  the  first  few 
values  of  j.  Parts  (a)  and  (b)  of  Figure  5.8  show  a  regular 
pattern  of  correlation.  This  is  further  evidence  of  the 
periodic  nature  of  the  interference  of  the  routing  update 
procedure . 
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Figure  5.9.  Pseudo  random  sequence  correlation. 


For  comparison  to  Figure  5.8  we  include  Figure  5.9 
which  shows  the  correlation  coefficient  computed  in  the  same 
way  for  a  sequence  of  pseudo-random  numbers.  Its  appearance 
is  much  like  Figure  5.8{c-f). 

With  the  argument  partially  sjoported  by  measure me nt 
and  simulation#  we  conclude  that  the  assumption  of  statisti¬ 
cal  independence  is  not  completely  unreasonable.  We  shall 
proceed  with  this  assumption  as  at  least  the  first  approxi¬ 
mation  to  the  behavior  of  the  actual  system. 

5. 3. 3. 3  Gag  E£Sbabiiiiy 

Having  stated  and  considered  some  of  the  implications 
of  the  assumptions  of  the  model#  we  are  now  prepared  to 
begin  the  analysis  of  that  model.  We  begin  by  finding  ex¬ 
pressions  for  the  probability  of  gap  occurrence  as  a  func¬ 
tion  of  the  delay  distribution  and  R ( D)  the  distribution 
which  characterizes  the  choice  of  the  destination  wait  value 
D.  Let  us  begin  with  method  E. 

5. 3. 3. 3.1  Gag  E£2b5bitit^  ffiC  BkSZba£k  mgttcd  £ 

Under  the  assumption  that  fill  and  network  delays  are 
each  independent#  we  may  find  the  orobability  with  which 
gaps  occur  in  the  output  process.  Let  y(i)  be  the  network 
aelay  and  f<i)  be  the  fill  time  of  message  i.  Let  t(i)  = 
y(i)  +  f(i>  -  y(i-1)  -  f(i-1)  be  the  difference  in  delay 
experienced  by  message  i  and  i-1.  Then  the  ft(i))  are  mutu¬ 
ally  independent  and  identically  distributed  random 
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variables  since  we  have  assumed  that  the  <y(1)>  and  'Cf(1)> 
are.  Let  the  nth  partial  sun  of  •Ct(i)>  be  P<n)  =  P(n-1)  ♦ 

t(n)/  and  let  P(0)  =  0.  Notice  that  P(n)  =  y(n)  +  f(n)  - 
Cy(0)  +f(0)D.  Now  let  us  define  the  strict  ascending  ladder 
indices  <N(k)>  for  tP(n)>  as  In  CPrab  653  (p.  140)  as  fol¬ 
lows: 

N(0) =0 

Nil)  =  minln  :  P(n)>0) 

N(k)  =  ninln  :  n>N(k-1)  and  P(n ) >P (N ( k -1 ) ) ) 

To  proceeds  we  need  the  following  theorem. 

Ih£S££lD.  A  gap  in  the  output  process  in  method  E  oc¬ 
curs  at  message  i  if  and  only  If  i  is  a  ladder  index  N(k) 

for  some  k>0  and  P<i)>D/  where  D  is  the  destination  wait 

time. 

An  output  pap  occurs  if  and  only  if  a  message 
arrives  after  it  is  needed  for  playout.  If  message  zero 
begins  filling  at  time  zero/^  then  the  time  at  which  message 
i  arrives  at  the  destination  is 

f(0)  +  f (1)  +  ...  +  f(i)  +  y(i) 

(i.e.r  the  time  to  fill  all  previous  messages  plus  the  time 
to  fill  message  i  olus  the  network  delay  of  message  i). 

Suppose  that  P(i)  <  0  for  all  i.  Then  for  each  i  we 

have 

P(i)  =  yli)  +  f(i)  -  (y(0)  +f(0>)  <  0  for  all  i.  (5.?) 

The  time  that  message  one  is  required  for  playout  is 

f(0)  +  ylti)  +  0  +  f(Q) 
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the  time  of  arrival  of  message  zero  plus  the  destina¬ 
tion  wait  time  plus  the  time  to  playout  message  zero).  From 
equation  5.2  we  have 

y(1)  +  f(1)  +  f(0)  <  y(0)  ♦  f(0>  +  D  +  f (0) . 

Therefore  message  one  is  not  late  and  does  not  cause  a  gap. 
Equation  5.2  may  be  rewritten  as 

fli)  +  y<i)  <  y<0)  +  f<0)  +  D. 

By  adding  f(0)  +  f(1)  +  ...  +  f(i-1)  to  both  sides  we  have 
i-1  i-1 

y'.  flj)  +  f(i)  +  y<i)  <  ^  f  ( i  )  +  ylO)  +  f(0)  +  D. 

j=0  j=0 

This  states  that  message  i  arrives  at  or  before  its  required 
time.  Hence  no  gaps  will  occur  when  P(i)  <  0  for  all  i. 

Suppose^  on  the  other  handi^  that  there  exists  an  i  such 
that  P<i)  >  0.  Choose  k1  to  be  the  minimum  such  i.  Then  k1 
is  a  ladder  index  by  definition#  since  it  is  the  first  oc¬ 
currence  of  a  value  of  P(i)  greater  than  D.  We  have 
f(0)  +  y(0)  +  0  <  f<k1)  +  y<k1) 
and  therefore 

k1-1  k1-1 

F(0)  +  y(0)  +  D  +  ^  f(j)  <  flj)  +  f(k1)  +  ylkl) 

j=0  j=0 

and  hence  the  first  gap  occurs  at  ladder  index  k1  whose 
value  Flkl)  is  first  to  exceed  D. 

After  the  occurrence  of  this  gap  at  k1#  message  k1  is 
output  immediately  upon  arrival  and  therefore  message  k1+1 
will  be  needed  at  time 
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2f<j)  +  y(k1)  +  f<lc1) 
3=0 


The  next  gap^  if  any#  will  occur  at  message  k?  where  k2  is 

the  minimum  i  for  which 

i  k1  i-1 

5^f(j)  +  y(i)  >  *  y(k1>  +  ^  fCj) 

j  =0  j  =0  3  =k  1 

(i.e.#  message  i  arrives  after  it  is  needed  for  playout). 
The  following  three  statements  are  each  equivalent  to  the 
last. 

f(i)  +  y(i)  >  f(k1)  ♦  y(k1) 

f(i)  +  y(i)  -  f(Q)  -  y(0)  >  f(k1)  +  y(k1)  -  y(0)  -  f(0) 
F(i)  >  Ptkl) 

Therefore#  since  k2  was  chosen  as  the  minimum  such  i#  k2  is 
the  ladder  index  immediately  following  k1 . 

We  may  now  prove  the  general  case  by  induction.  Assume 
that  the  first  n-1  gaps  occurred  at  the  first  n-1  ladder 
indices  whose  values  are  greater  than  D.  Let  I  be  the  last 
ladder  index  at  which  a  gap  occurred.  After  the  occurrence 
of  this  gap  at  I#  message  I  is  output  immediately  upon  ar¬ 
rival  (as  was  message  k1)  and  therefore  message  1+1  will  be 
needed  at  time 
I 

^  f  (j)  +  y<  L)  +  f ( I ) 
j=0 

The  next  gap#  if  any#  will  occur  at  message  It  where  11  is 
the  minimum  i  for  which 


i  I  i-1 

53f(j)  +  y(i)  >5Zf(j)  +  y(l>  * 
j=0  i=0  j=l 

message  1  arrives  after  it  is  needed  for  playout). 
Once  again^  the  following  three  statements  are  each 
equivalent  to  the  last. 

f(i)  +  y(i)  >  f(l)  +  y(l) 

f(i)  +  y(i)  -  f(0)  -  y(0)  >  f(l)  +  y(l)  -  y(C)  -  f(0> 
P(i)  >  P(l) 

Hence  11  is  the  ladder  index  immediately  following  1. 
Therefore  each  ladder  index  whose  corresponding  value  is 
greater  than  D  has  an  associated  gap;  and  a  gap  occurs  only 
at  a  ladder  index  whose  corresponding  value  is  greater  than 

D. 

G.£. D. 

Thus  the  probability  that  the  kth  message  in  a  sentence 
produces  a  gap  is  equal  to  the  probability  that  k  i s  a 
ladder  index  and  P(k)  >  i.e./^ 

PrCgap  |  k^DD  =  PrCk  is  a  ladder  index  and  P(k)>D!) 
According  to  CPrab  65]  (p.  141)  we  have 

PrCgap  |  k,D]  =  PrCP<k)>0/^  P  (k-1  )  >0/- .  .  . P(1)>0/  and 
P{k)>DJ 

Since  P(k)>D  implies  P(k)>0  and  P(1)  is  guaranteed  to  be 
greater  than  zeror  we  have 

PrCgap  1  k/DD  =  PrCP<k)>0/  P  (k-1  )>0^  .  .  . P(c)>0] 

Assume  that  the  (fill  plus  network)  delay  is  distributed 
with  probability  distribution  function  S(y)^  that  the  delay 
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of  the  first  tnessaqe  in  the  sentence  is  t#  and  that  D  is  the 
destination  wait  time.  Then 


k-2 

PrCgap  |  k^t/DD  =  Cl -S (t  +  D)D C 1-S  ( t  )] 


We  may  remove  the  condition  on  k  to  obtain  the  probability 
that  a  gap  occurs  at  a  randomly  selected  message  in  a  sen¬ 
tence  of  length  n^  given  t  and  D#  i.e./ 

1  n  k-2 

PrCgap  |  ti-D^n:  =  -  C1-SCt+D) D  C1-S  (t  >3 

n  k  =  2 


Removing  the  condition  on  t  we  have 

1  n-2  ^00  k 

PrCgap  |  Dm]  =  -  2^  /  C1  -  S(  t  +  D  )D  C 1 -S  ( t )  3  dS(t) 

n  k=C  Jt=0 


Removing  the  condition  on  D  we  have 

1  n-2  /*  00  ^00  k 

PrCgap  /n3  =  -  /  /  Cl -S( t+D )3 Cl -S ( t ) 3  dS(t)dR(D) 

n  k  =  0  -'D  =  U  •'t=0 

(5.3) 


where  R(D)  is  the  distribution  fun ct ion  of 
wait  time  D.  Since 
n 


i 

X 


1-x 

1-x 


the  destination 


we  have 


PrCgap  /n3 


C1-S(t+0)3 


n 

1-C1-5lt)3 

S(t) 


-1 


dS(t)dR(D) 

(5.4) 
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ye  shall  evaluate  this  expression  nuserically  in  succeeding 
sections. 

5. 3. 3. 3. 2  tSC  playback  selbQd  I 

Since  there  is  no  expansion  of  the  tiee  axis  in  method 
If  ye  expect  that  when  a  Message  exceeds  the  delay  of  the 
first  message  by  more  than  D  a  gap  results.  The  destination 
arrival  time  of  message  k  is  (as  before)  oiven  by 
k 

Zf(i)  +  y<k) 

j=o 

The  time  at  which  message  k  is  required  is 

k-1 

f(0)  ♦  y<0)  +  D  ♦  52  f  <  j) 

j  =  0 

Hence  we  have  a  gap  whenever 

k  k-1 

Vtlj)  ♦  y<k)  >  f<0)  +  y<0)  +  0  + 
i=0  7=0 

or 

f<k)  +  y(k)  >  f<0)  +  y<0)  +  0. 

Therefore  the  probability  of  a  gap  at  message  k^  given  that 
f(0)  +  y(0)  =  t  and  D  is  the  destination  wait  time^  is  1- 

S(t+D).  For  sentences  of  length  n  messages  we  have 

1  n-1  n-1 

PrCgap  |  t^O/n]  =  -  3  =  -  C1-S(t+D)3 

n  k=1  n 

Removing  the  condition  on  t  we  have 
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n-1  ^00 

PrIIgap  |  D^n3  = -  /  [1-S(t*-D)D  dS(t) 

n  Jt  =  0 

Removing  the  condition  on  0  we  have 

n-1  ^00  foo 

PrCgap  |  nD  =  -  I  /  C1-S(t+D):  dS(t)dP(D)  (5.5) 

n  Jo=0  Jt=0 

Comparing  the  right  hand  side  of  this  to  that  of  equation 
5.3  we  notice/  as  expected/  that  this  expression  is  larger. 
Therefore/  in  general/  the  gap  probability  is  lower  in 
method  E  than  in  method  I.  Numerical  results  appear  below. 

5. 3. 3. 4  Dgia^  BCfiditlsrS 

In  the  previous  two  sections  we  have  assumed  a  distri¬ 
bution  for  the  destination  wait  time  h .  The  purpose  of  this 
section  is  to  explore  some  possible  delay  predictors  each 
yielding  a  distribution  function  for  D.  Three  schemes  are 
discussed  and  all  are  based  on  monitoring  which  may  be  per¬ 
formed  at  the  destination  node.  The  first  scheme  predicts 
delay  variation  by  computing  the  range  of  delay  for  previous 
samples.  We  have  found  that  this  scheme  "learns"  quickly 
tut  performs  poorly  when  occasional  long  delays  occur.  In 
order  to  ignore  these  "spikes"  in  delay/  and  thus  achieve 
smaller  D/  we  introduce  the  second  scheme  which  views  only 
that  portion  of  the  range  below  a  threshold.  The  third 
scheme  attempts  to  track  the  chanoinq  delay  by  modifying  an 
internal  counter  to  predict  message  arrival  times. 

The  basic  idea  then/  in  each  scheme/  is  to  measure  the 
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delay/  and  based  on  the  information  gained/  to  make  some 
intelligent  choice  for  h  at  each  sentence  boundary.  It  is 
clear  that  the  larger  is  D  the  fewer  are  the  gaps  which  will 
occur  in  the  output  stream.  However/  large  D  tends  to  des¬ 
troy  the  interactive  nature  of  the  communication.  Ideally  a 
scheme  would  "optimally"  balance  these  two  properties.  It 
is  difficult  to  define  optimality  here.  However/  the  fol¬ 
lowing  general  statement  holds:  Large  destination  wait  time 
and/or  frequent  gap  occurrence  each  yield  a  poor  quality  of 
communi cat  ion. 

5. 3. 3. 4.1  m-saaelg  rangg 

In  both  playback  methods/  if  the  delay  of  a  message 
within  a  sentence  exceeds  the  deley  of  the  first  message  of 
that  sentence  by  more  than  D/  then  at  least  one  gap  will 
occur.  The  total  range  of  the  previous  m  (m>1)  samples  is  a 
pessimistic  estimate  for  the  difference  between  maximum 
delay  within  a  sentence  and  the  delay  of  the  first  message 
in  that  sentence.  It  is  pessimistic  because/  or  the  aver¬ 
age/  the  first  delay  will  fall  somewhere  between  the  ex¬ 
tremes.  The  problem  is  to  find  the  distribution  of  the 
range  of  m  samples.  Our  derivation  below  follows  that  of 
CGumb  673  <pp.  97-98).  We  begin  with  the  joint  probability 
that  t  is  the  minimum  (denoted  t=iiin)  and  t  +  z  is  the  maximum 
(denoted  t-»'Z  =  max)  among  m  samples  each  drawn  independent  ly 
from  distribution  S(y). 


PrCt=inin#  t-*-z=nax  |  m  samples! 


=  PrCat  least  one  sample  =  tr 

at  least  one  sample  =  t+z/  anri 

m-2  samples  fall  in  the  interval  Ct/t+z!  ! 

ir-2 

=  m  dS(t)  (m-1)  dS(t+z)  [ S ( t+z ) -S (t ) ! 


By  removing  the  conditions  on  t  and  t+z  we  find  that  the 
distribution  of  range  R(0  j  m)  =  PrCrange<h  |  m  samples!  is 
given  by 


R(D  I  m) 


m  (m-1)CS(t+z)-S (t)! 


m-2 


dS ( t  +  z ) 


dS(t) 


Yielding 


R([>  I  m) 


t  00 

ml  CS(t+P)-S(t) ! 


ip-l 


dS(t) 


(5  6) 


This  equation  cannot  be  reduced  further  without  knowledge  of 
S(t)  . 


5. 3. 2. A. 2  m-jamaig  BiCtiSi  caDSfi 
The  full  range  estimate  functions  fairly  well  (as  we 
show  later)  but  it  is  not  without  fault.  Since  the  entire 
range  is  used/  isolated  cases  of  high  delay  cause  an  unduly 
large  value  to  be  chosen  for  the  next  destinatioi  wait  time. 
For  this  reason  we  wish  to  examine  the  partial  range  of  m 
samples  where  the  k  highest  values  are  ignored  (m>1  and 
k<ffl-1).  We  wish  to  find/  as  before/  the  distribution  R(h  | 


■/k)  of  the  partial  range  given  the  distribution  S(y}  from 
which  the  m  samples  are  chosen. 

We  begins  as  before#  with  the  joint  probability  that  t 
and  t+z  are  among  the  m  samples  drawn  from  S(y)#  where  t  is 
the  minimum#  and  t-*-z  is  the  maximum  among  the  m-k  smallest 
samples. 

PrCdrawing  such  a  samplel 

=  PrCat  least  one  sample  =  t# 

at  least  one  sample  =  t*Zf 

m-k-2  samples  are  in  [t#t+2D#  and  there  are 
k  samples  which  exceed  t-<-z  D 

m-k-2 

=  mdS(t) (m-1)dS<t+z) CS<t+z)-S(t>:  ^  jC1-S(t+z)D 


Therefore 


R(0  I  m#k) 


m(m-1 ) 


m-k-2  k 

CSlt+z)-S(t)]  C1-S(t+z):  dS(t+z)dS(t) 

(5.7) 


5. 3. 3. 4. 3  A  dgia^  ££a£k££ 

A  third  method  of  delay  prediction  was  developed  by 
James  Forgie  CForg  76a3  of  the  ^Massachusetts  Institute  of 
Technology  Lincoln  Laboratory#  Lexington#  Massachusetts.  It 
compares  the  time  stamp  of  an  arriving  message  with  the 
"arrival  clock"  and  adjusts  that  clock  bv  plus  Cor  minus) 
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one  count  if  the  message  arrived  earlier  (or  later)  than 
predicted  by  the  clock  value.  At  the  beginning  of  a  sen¬ 
tence  a  quantity  called  the  "reconstitution  delay"  c  is  sub¬ 
tracted  from  the  arrival  clock  to  set  the  playout  clock.  It 
is  important  to  note  that  the  value  of  c  is  chosen  at  the 
beginning  of  a  conversation  by  the  user.  The  waiting  packet 
begins  playout  when  the  playout  clock  reaches  the  value  of 
the  time  stamp  within  the  packet.  Time  is  measured  in  units 
equal  to  a  frame  siae  (Id. 2  msec  for  Linear  Predictive  Cod¬ 
ing#  LPC#  algorithm  used  experimentally  for  speech  communi¬ 
cation  in  the  ARPANET#  see  CCohe  763)  and  the  clock  is  in¬ 
creased  by  one  at  each  frame  time. 

Figure  5.10  illustrates  the  operation  of  the  algorithm. 
The  first  message  of  the  sample#  over  which  the  arrival 
clock  a(x)  is  adjusted#  arrives  at  the  destination  and  ini¬ 
tiates  the  clock  value  (assumed  to  be  at  time  zero).  The 
time  stamp  in  each  message  is  shown  as  the  ordinate  of  a  dot 
at  the  point  (x#y)#  where  x  is  the  arrival  time  of  the  mes¬ 
sage  at  the  destination  and  y  is  the  time  stamp  within  that 
message.  Notice  that  an  "early"  arrival  has  its  time  stamp 
above  the  arrival  clock  and  a  "late"  arrival  appears  below. 
The  arrival  clock  therefore  attempts  to  "track"  the  points 
(x#y)  by  adding  one  at  points  where  y  >  a(x)  and  subtracting 
one  where  y  <  a(x).  Were  the  clock  unadjusted#  the  value 
would  be  floor(x)  (the  largest  integer  less  than  or  equal  to 
x).  Let  the  clock  difference  l(x)  be  defined  to  be  a(x) 
floor(x).  Then  a(x)  predicts  the  time  stamp  of  a  message 
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arriving  at  time  x.  It  therefore  predicts  that  a  message 
arriving  at  time  x  with  time  stamp  y  should  (have)  arrive(d) 
at  X  -  (a(x)  -y);  or  more  precisely  f loor (x-a(x)-y) .  Since 
a(x)  and  y  are  integers  we  have  that  the  predicted  arrival 
of  a  message  arriving  at  time  x  with  time  stamp  y  is  y  - 
l(x)»  At  the  arrival  of  message  m  (the  start  of  a  sentence) 
the  playout  clock  p(xm)  !s  set  to  a(xm)-c/  and  then  steps  at 
frame  boundaries.  Playout  begins  when  p(x)  >  ym  or 
equivalently  r  units  after  the  predicted  arrival  time  of 
message  m  (i.e.x  at  time  ym  -  l(>m)  +  r).  Hence  for  the 
value  0  as  previously  defined  we  have  D  =  ym  -  l(xm)  +  c 
-xm.  We  shall  disallow  negative  D  and  thus  we  write 

0  =  max-CO^  c  +  ym  -  xm  -l(xm)>.  (5.8) 

This  algorithm  may  be  modeled  by  a  Markov  chain 
CKlei  75]  where  the  states  represent  the  clock  difference 
value  l(x)  and  transitions  occur  at  message  arrival  points. 
When  a  message  arrives  with  delay  less  (more)  than  predicted 
by  the  current  state  k/^  then  a  transition  to  state  k  +  1  (k-1) 
occurs.  As  the  function  l(x)  starts  at  value  zero  the  Mar¬ 
kov  chain  will  start  in  state  zero.  Suppose  t  is  the  delav 
of  the  first  message  (i.e.^  that  message  which  initiates 
arrival  clock).  Given  that  the  current  state  is  Vr  the  oro- 
bability  of  a  transition  from  state  k  to  k+1  is  equal  to  the 
probability  of  experiencing  a  delay  less  than  t  +  kf/  where 
f  is  the  frame  size.  Likewise^  the  probability  of  a  transi¬ 
tion  from  k  to  k-1  is  equal  to  the  probability  of  a  delay 
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greater  than  or  equal  to  t  -  Otherwise/^  we  reaain 
in  state  k  if  the  delay  lies  between  t  -  kf  and  t  -  (k-1)f. 
Figure  5.11  displays  this  graphically  for  the  continuous 
distribution  function  S<t). 


S(t-(kHf»  S(t-kf) 


Figura  5.11.  May  trackar  Markov  chain. 

These  transition  probabilities  insure  the  existence  of 
an  irreducible  (sub-)  Markov  chain.  If  the  random  variable 
is  bounded  then  the  chain  will  have  a  finite  number  of 
states.  Since  the  chain  is  irreducible  we  may  equate  the 
flow  into  and  out  of  state  k-»^1  as  follows: 

flow  in  =  flow  out 

p(k) CS(t-kf)3  +  p(k+2)C1-S(t-(k+1)f)3 

=  p(k+1)C1-S(t-kf)3  *  p(k+1>tS(t-(k+1)f)3 
where  p(k)  is  the  steady-state  probability  of  being  in  state 
k  (we  have  assumed  that  S(t)  is  continuous). 

Solving  for  p(k)  we  have 

p(k+1)C1-S(t-kf)+S(t-(k+1)f )3  -  p(k+2)C1-S(t-(k+1)f )3 


S(t-kf > 


Equation  (5.9)  defines  a  recursive  procedure  for  calcu¬ 
lating  the  steady-state  state  probabilities  for  the  f'arkov 
chain.  Unfortunately#  in  the  current  form  it  requires  an 
infinite  number  of  steps  in  general.  One  can#  however# 
achieve  an  approximation  using  the  following  general  proper¬ 
ties.  There  exists  some  I  for  which  SCt-d  +  Df)  =  0  and 
S(t-Lf)  >  0  (i.e.#  the  random  variable#  delay#  is  bounded 
below).  For  a  random  variable  which  has  a  finite  lower 
bound  and  finite  mean  value  we  have  that)  for  any  e  >  P  we 
may  find  a  finite  positive  integer  I*  such  that  1  -  S(t+l*f) 
<  e.  Therefore  the  steady-state  of  being  in  any  state  num¬ 
bered  -L *-1  or  less  is  less  than  e.  If  L=l+l’+1#  we  need 
only  perform  L  steps  (plus  the  usual  normalization  step)  to 
achieve  a  value  for  the  state  probabilities  each  within  e/L 
of  the  correct  value. 

Thus  far  we  have  considered  only  the  steady-state  pro¬ 
babilities.  One  may  also  wish  to  consider  the  m-step  state 
probabilities  given  that  we  begin  in  state  zero.  Let  p(k 
I  m#0)  be  the  probability  of  being  in  state  k  after  m  transi¬ 
tions  starting  from  state  zero.  Then 

p(0  |0#0)  =  1 

p(k  I  l#0)  =  p(k-1  I  l-1#0)CS(t-(k-1) f)D 

♦  p(k  I  l-1#0)CS(t-(k-1)f)-S(t-kf ):? 

♦  p(kel  I  l-1#0)C1-S(t-kf): 

This  defines  another  recursive  procedure  which  terminates 
after  m  iterations  for  at  most  Zm+I  states. 

cy  computing  the  state  probabilities  as  indicated  we 
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find  the  distribution  function  R<D).  Note  that  the  proba¬ 
bility  that  l(x)  =  k  is  just  p(k).  Assumino  th&t  a  aessage 
arrives  to  start  a  sentence  at  time  x  with  time  stamp  y  then 
from  equation  5.B  we  have 

R<D  I  x#y^c)  =  PrCma xCO/C+y-x-l (x ) >  <  0  D 

=  PrC  c+y-x-l(x)  <  D  3 

=  PrC  l(x)  >  c+y-x-D  3 

1+1 

R(0|x#y^c)=  5^  p(i) 

i  =  f loor<  c  +  y-x-0) 

The  quantity  x-y  is  the  message  delay  (both  fill  and  net¬ 
work)  which  we  have  earlier  assumed  to  be  distributed  as 
5>(t).  Therefore 

/“  00  1*1 

R(D  I  c)  =  /  2}  p(i)  dS(t) 

Jt-0  i=f loor<c-t-D) 

We  s^all  not  evaluate  this  numerically/  but  later  show  some 
simulation  results  in  the  use  of  this  algorithm. 

we  have  assumed  a  constant  c  in  the  analysis  thus  far 
and  this  was  the  case  in  the  actual  implementation  of  the 
algorithm  at  Lincoln  Laboratory  (i.e./  constant  for  the 
length  of  a  conversation).  An  automatic  method  of  adjusting 
c  to  the  network  dynamics  has  been  developed  (again  by  James 
Forgie).  After  an  initial  value  is  given  to  c/  it  is  in¬ 
creased  if  a  gap  occurs  and  decreased  if  a  strino  of  s 
(currently  contemclated  to  be  ICO)  messages  pass  through  the 


system  without  gaps.  This  may  again  be  modeled  by  a  r^arkov 
chain. 

The  new  chain  will  be  three  dimensional  with  states 
indexed  by  k  the  old  state  labels^  i  the  current  length  of 
the  gapless  string^  and  the  current  value  of  c.  Let 
GCk^i^c)  =  PrCgap  occurs  given  l(x)=k^  gapless  string  length 
=  i/  and  c  =  the  reconstitution  delayD.  Transitions  occur 
at  message  arrivals  as  before.  It  is  simpler  to  visualize 
transitions  as  taking  two  steps.  First  consider  the  part  of 
a  transition  which  is  governed  by  the  gap  probabilities. 
The  possible  transitions  and  their  probabilities  are  as  fol¬ 
lows 

PrC(k^i#0)  to  (k,i  +  1,0)l  =  1  -  G(k/>i/.0) 

PrClk^S/c)  to  (k^0rc-1)3  =  1  -  Glk/^s^c) 

FrClk^i^c)  to  (k^i+1»c)D  =  1  -  G(k/i/c) 

PrCCk^i/c)  to  (k/0/C+1)D  =  G<k#i/c) 

Next  consider  the  part  of  a  transition  due  to  a  change  in 
delay. 

PrC(kri^c)  to  (k^i^c)]  =  S(t-(k-1)f)  -  S(t-kf) 
PrC(k^i^c)  to  (k+1^i#c)3  =  S(t-kf) 

PrC(k^i/c)  to  (k-1/i/c)3  =  1  -  S(t-{k-1)f) 

We  attempt  no  further  solution  to  this  problem^  but  rather^ 
we  later  present  simulation  results. 
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5. 3. 3. 5  CaiiUQicAlifiQ 

Before  proceeding#  we  need  a  basis  for  comparing  the 
"quality"  of  communication  in  our  stream  traffic  context. 
Among  the  parameters  affecting  the  quality  of  communication 
in  the  system  under  discussion  are  G#  the  probability  of  an 
output  gap  occurrence  and  f+t+D#  the  speaker-to- I i st ener 
(end-to-end)  delay.  The  end-to-end  delay  and  the  gap  proba¬ 
bility  are  both  influenced  by  the  destination  wait  time  D. 
Ideally  one  would  choose  to  operate  at  the  point  D=0  and 
6=0.  Unfortunately  this  cannot  be  ach’eved  in  general. 
There  is  a  tradeoff  between  these  two  parameters.  Experi¬ 
ence  has  shown  that  we  can  tolerate  nonzero  6  up  to  some 
threshold  value  CForg  763  beyond  which  the  quality  degrades 
rapidly.  For  the  experiment  reported  in  CForg  763  roughly  6 
percent  of  the  packets  caused  gaps  and  the  quality  was  con¬ 
sidered  to  be  acceptable.  It  is  likely  that  a  similar 
behavior  is  true  for  0  (i.e.#  beyond  some  value  of  f+t+D  the 
communication  ceases  to  be  of  an  interactive  nature).  Stu¬ 
dies  regarding  the  effect  of  transmission  delay  on  telephone 
conversations  reported  in  CKrau  673  and  CKlem  673  show  that 
the  threshold  is  above  .6  sec  and  below  l.fi  sec. 

we  wish  to  choose  a  quality  function  dlh/'G)  which  has 
this  threshold  behavior.  One  such  family  of  functions  is 

1/(2n-1) 

1  +  (1-y) 

-  y<2 

q(y)  =  2  (5.10) 

0  y>2 
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;  t,  . 


Soae  Members  of  this  faHlly  are  shown  in  Figure  5.12 


Suppose  we  define 
Q(D/0)  =  q(D/d)  and 

Q(Q^G)  =  q(G/q)  for  some  given  thresholds  d  and  g. 

It  remains  to  fill  in  the  DG  plane  so  as  to  continuously 
coincide  with  the  axis  functions.  As  mentioned  earlier^  the 
point  of  highest  quality  is  the  origin.  Then  quality  should 
oecrease  as  we  increase  the  distance  from  the  origin.  One 
reasonable  choice  for  Q(D#G)  is  then 


Another  choice  would  be  the  product  of  the  axis  qualities 

i .  e.  # 

alD^G)  3  G(0^0)Q(0/G). 

Any  quality  function  will  define  contours  of  constant 
quality.  We  shall  define  the  region  of  acceptability  to  be 
the  region  enclosed  between  the  positive  D  and  6  axes  and 
the  DG  plane  projection  of  the  contour  whose  value  is  G(d^O) 
=  Q(u^g).  This  region  is  independent  of  n  and  an  example  is 
shown  in  Figure  5.13.  Other  contours^  however  increase  in 
distance  from  the  origin  with  increasing  n.  This  property 
is  illustrated  in  Figures  5.14  and  5.15.  Therefore  by  ad¬ 
justing  n  we  may  change  the  importance  of  the  origin  as  the 
optimal  operating  point.  If#  for  example^  one  wishes  to 
assign  the  same  value  of  quality  to  all  points  in  the  region 
of  acceptability  then  an  infinite  value  of  n  should  be 
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Figure  S.1 2.  One  paramettr  quality  functions. 


Gap  probability  Gap  probability 


Mean  destination  wait  time. 


Figure  5.13.  Region  of  acceptability. 


Figure  5.14.  Contours  of  quality  .9. 
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Matn  destifMtion  wait  tima 
Figura  5.15(b).  Coratant  quality  oontoun  (n  •  10). 
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chosen.  With  communication  quality  defined  by  equations 
5.10  and  5.11  we  may  proceed. 

5. 3. 3. 6  Nuffierisai 

The  purpose  of  this  section  is  to  present  some  numeri¬ 
cal  results  by  substituting  a  delay  distribution  into  the 
formulas  derived  in  the  preceding  sections.  Exact  results 
will  be  given  for  an  exponential  distribution  of  delay.  For 
the  more  general  class  of  r-stage  Erlang  distributions  nu¬ 
merical  integration  is  used. 

5. 3. 3. 6.1  Ihg  exeangntiti.  di  stgifeu  tifin  of  dgia^ 

For  the  purpose  of  demonstration/  we  choose  the  ex¬ 
ponential  distribution 

-t/  X 

S(t)  =  1  -  e 

Admittedly  this  is  a  poor  choice  for  the  distribution  of 
network  delay!  The  measurements  presented  earlier  substan¬ 
tia*®  that.  This  distribution/  however/  has  a  large  coeffi¬ 
cient  of  variation  when  compared  to  the  measured  distribu¬ 
tions.  It  therefore  provides  a  lower  bound  on  performance/ 
which  should  hold  for  less  tractable  but  more  realistic  dis¬ 
tributions.  Some  r-stage  Erlang  distributions  (more  what 
one  expects  from  a  network)  are  treated  in  the  next  section. 

Since  R(0)/  the  distribution  of  destination  wait  time/ 
is  required  in  the  gap  probability  formulas/  we  shall  first 
find  R<D  I  m/k).  We  begin  with  equation  5.6.  Substituting 


for  S(t)  we  have 


•'t  =  0 


]m-1 

1  -t/  X 

-  e  dt 


m-1 


r 

^00  1  -mt/x 

=  m  1-e 

/  -  e  dt 

L  J 

-'t=U  X 

IP-1 


R(D  I  in) 

The  mean  value  of  D  given  m  is  given  ly 


C(m)  = 


m-? 

m~1  /*°°r  -D/xj  -O/x 

=  -  /  0  1-e  J  e  dD 

X  Jt>=0 


ir-l  m-2  i  /m-2\  ^  oo  -(i  +  1)i>/x 

-  S  (  )  I 

X  i=0  V  i 


IP-1  m-2  1  /ni~2\  x 

=  —  E<-1>  (  ) - 

X  i  =G  \  i  /  (  i  +1  )  ( •» 


+1 ) 


=  X 


m-l  i-1  /in~1\  1 
i=1  \  i  /  1 


m-2  i-1  1  m-1  i-1  /m- 

( J;  *  (,. 


X  m-1 

B(m-1)  +  -  1-(1-z) 

m-1 


2  =  1 


(5.12) 


1P6 


5  .7 

S  (0 


D(m)  =  D (m-1 )  ♦  - 

m-1 


Therefore^ 


tn-1  1 
2(in)  =  X  - 

1  =  1  i 


(5.13) 


K'ext  we  evaluate  R(D  |  m^k).  Beqinning  with  equation 
and  substituting  for  S(t)  we  have 
rti-2 


D/k)  =  m(m-1) 


f  f  t 

7t  =  C  -/z=0 


ID-  k-  ? 

-t/x  -(t+z)/x')  -k{t  +  z)/x 


e  -e 


J 


1  -(t+z)/x  1  -t/x 

-  e  -  e  dz  dt 

X  X 


ni-2  1 


=  Tl  (W-1  ) 


-f 

X  A=o 


-mt  /  X 

e  dt 


m-k  -2 


-(k+1)z/x 
e  dz 


(in-2\  ni-k-2  i  /m-k-2\  fX>  -(i+k  +  1)z/x 

.  )  *?o  <■”  (  ,  )X=0  ' 


/ir.-2\  m-k-2  i  /m-k-2\  1-e 

=  «<”-’>(  ,  )  ,i;  <-'>  (  ,  ) 


-(i  +  k  +  l  )D/x 


R ( D  I  mr  k) 


i  +  k  +  1 


Suppose  k  =  0. 


-(i  +  1  )P/x 

ir.~2  i  /n-?\  1-e 

R(0  I  ir.^O)  =  xCm-D  ^  (-1)  (  )  - 

i=0  \  i  /  1+1 


(  J  ’ 


m-1 

1-(1- 7) 


m-1 

1-(1-7) 


-D/x 

2  =  e 


R(D  I  m/G) 


m-1 


which  agrees  with  equation  5.12. 

It  is  useful  to  check  to  see  that  R(n  |  m/k)  has  the 
proper  characteristics  of  a  probability  distribution  func¬ 
tion.  Clearly  R(u  |  m/k)  =  0  (as  desired).  Let  us  show 
that  the  limit  of  R(D  |  m/^k)  as  D  approaches  infinity  is 


one . 


limit  ft  ( D 
D 


I  m^k )  =  ( m 


(m-2\  ii-k-2  i  /m-k-2\  1 

)  E  (  )  — 

k  /  i=n  \  i  /  i+k+1 


m-2\  in-k-2  i  /m-k-2\  (i  +  1  )  (i  +  2)  . . .  (  i  +  k) 


=  (m 


(m-2\  in-k-2  i  /m-k-Z\ 

)  Z  (-1)  (  )  - 

k  /  i=C!  V  i  /  (i 


+1)(i+2)...(i+k) (i+k+1) 


1  ir-1  i-k-1  /"i-lv 

=  -  Z  I  )(i-1>(i-2>...(i- 

k!  i»k+1  \  i  / 


k) 
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k  k  «-1 

(-1)  d 

k!  k  z 

dz 


Therefore  P(0  |  m^k)  has  the  proper  limiting  value.  For  any 

finite  D#  we  have 

- Ci+k+1)0/x 
1-e  <  1 


Hence  MD  |  *^k)  <  1  and  thus  has  the  properties  of  a  proba¬ 
bility  distribution  function.  It  remains  to  find  an  expres¬ 
sion  for  the  mean  value  of  0  given  m  and  k.  We  hegin  with 
the  usual  formula. 


fee 

k)  =  /  0 

-'0  =  0 


dR<D  I  m^k) 


Substituting  for  9(0  |  m/k)  we  have 

/m-2\  m-k-2  i  /m-k-2> 


/m-2\  m-k-c  i  /m-x-Cx  /-qo  p 

=  (m-l)/  ]  Z  (-1)  (  )/  - 

\  k  /  i=0  \  i  /./D=0  X 


0  -(i+k+1)P/x 


(ir-2\  ir-k-2  i  /m-k-2\ 


(i  +  k  +  1) (i  +  k  +  1  ) 


Ti-1  i-k-1  \  / ^ ~^ \  ^ 

h(m»k)  =  X  ^  (-1)  (  ){  )  - 

i=k  +  1  \  i  /  \  k  /  i 


(5.15) 


In  order  to  simplify  this  further  let  us  consider  the  fol- 
lowino  difference 
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Beginninp  with  equation  5.17  and  perforaing  repeated  appli¬ 
cation  of  3.16  we  have 

2(k  +  3/-k)  =  D(k  +  2^k)  +  x/(k  +  2) 


^(m#  k ) 


m-l  1 

X  ^ 
i=k+1  i 


From  equation  5.13  it  follows  that 

Dliiiok)  =  fi(m^O)  -  0(k+1^0)  (5.18) 

Figure  5.16  shows  the  behavior  of  0{in#k)  for  m  from  2 
to  10U  and  several  k.  The  k=0  curve  is  logarithmic  in 
shapes  having  its  most  rapid  ascent  at  the  beginning  and 
then  tapering  off.  This  is  a  useful  property  in  the  selec¬ 
tion  of  both  m  and  k.  The  difference  between  D(m/0)  and 
0<m-1/0)  '  .e./  D=1/(m-1)  )  is  the  curve  from  which  the 
0(n#k)  curves  emanate.  This  shows  that  the  difference 
between  successive  values  of  D(m^k)  is  very  small  beyond  say 
ni=20 . 

Let  GEln^m^k)  be  the  gap  probability  given  that  (1) 
each  sentence  contains  n  messages^  (2)  D  is  the  partial 
range  of  m  samples  with  the  k  largest  samples  discarded^  and 
(3)  the  delay  distribution  is  exponential.  Initially  we 
shall  drop  the  m  and  k  parameters.  Then  equation  5.3  yields 
the  following 
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Figur*  S.16.  Maan  ranga  vs.  tampl*  siza  for  tha  axponantial  distribution. 
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1  r  CO  feo 

6E(n)  =  -  /  / 

n  1=0  yD=o  yt=o 


-t/x  -0/x  -ti/x  1  -t/x 
e  e  e  -  e  dt  dR(D) 

X 


1  n-2  y^oo  “O/x  ^  oo  1  -t(i+2)/x 

=  -^/e  /  "®  dR(D) 

n  1=0  *'D=0  Jt-0  X 


1  n-2  /*oo  -D/x  1 

=  -  53  /  ®  -  dR(D) 

n  i=0  •'0=0  i*-2 


(5.10) 


Suppose  k=0.  Then 


m-2 


1  n-?  1  /*  00  -O/xT  -O/x"!  m-1 

GE(nym^O)  =  -  JZ  -  /  ®  I"®  - 

n  i=0i  +  2*'n  =  0  L  X 


-O/x 

e  do 


n-1  n-2  1  III-2  j  /'»-2\  1  /•oo  -D(j+2)/x 

-  D  -  E  (-1)  (  -  /  e  dO 

n  1  =  0  1+2  j=0  \  j  /  *  •'0=0 


m-1  n-2  1  m-2  i  /m-2\  1 

=  —  E  —  E  (-1)  (  )  — 

n  1=0  1+2  j=0  \  3  / 


1  n-2  1  m  3/™\ 

=  —  E  —  £  I 

mn  i=0  1+2  i  =2  \j  / 


1  n-2  1  d 

—  E - 

mn  1=0  1+2  dz 


(1-z)  -1 


Z  =  1 


1  n-2  1 

GE(nym/C)  =  '  ^3 

mn  1=0  1+2 


(5.20) 


This  function  decreases  as  m  or  n  increase  (for  n>2). 
Figure  5.17  shows  a  family  of  m  versus  GE  for  n  =  ?,  l,  R, 
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16^  24/  32 


Each  curve  falls  rapidly  initially  and  then 


flattens  out  as  tn  increases. 

For  oeneral  k  we  substitute  the  right  hand  side  of 

equation  5.14  into  5.19  to  obtain 

1  n-2  1  /•qo  -h/x  m-2 

GE(n/m/k)  =  -  ^  -  /  e  (m-1) 

n  i=0  i  +  2  ./o  =  0  k 

in-k-2  j  /m-k-2\  1  -(i  +  k  +  1)D/x 

^  (-1)  (  -  P  (5.21) 

j=0  \  i  /  X 


m-1  m-2  n-2  1  m-k-2  j  /m-k-?\  1  /* 

—  E  —  E  (-1)  (  -  / 

n  k  1=0  i+2  i=0  \  j  /  x 


QO  -(j+k  +  ?)h/x 
e 

-D 


ffl-1  m-2  ri-2  1  m-k-2  j  /m-k-2\  1 

~  E  —  E  <-i>  (  ) - 

n  k  i=0  i+2  j=0  \  i  /j+k+2 


1  n-2  1  m  j-k-2  /m\  (j-D! 

—  E  —  E  <-■•>  ) - 

mnk!  i=0  i+2  j=0  \j/  (j-k-2)! 


k  +  1  tr-1 

1  n-2  1  k+1  d  1-(1-z) 

—  Y,  —  (-1)  -  - 

ronx!  i=0  i+2  k+1  ^ 

d? 


z=1 


k+1 

n-2 

1 

GE(n^in^k}  = 

E 

=  (k  +  1 )  GE  (n/m/0) 

(5.22) 

mn 

i=0 

i  +  2 

Figure  5.18  shows  GE  versus  m  for  n=l6  and  several  k. 


dD 
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GE<16.m,k)  (%) 


We  define  Gl(n^m^k)#  the  gap  probability  for  method  Ir 
similarly  to  GE(n^ffl/k}.  Beginning  with  equation  5.5  we  have 


n-1  r  00  r  CO  -t/x  -D/x  1  -t/x 

GI(n)  =  -  11^  ®  ®  dR(D) 

n  J{>=0  Jt  =  0  X 


n-1  ^00  -D/x  r  oQ  1  -?t/x 

=  -  /  ®  /  ■  ®  dR(D) 

n  yD=0  Jt=0  X 


n-1  ^  a 

2n  A={ 


n-1  /“oo  -h/x 

e  dR(D) 

=  0 


(5.23) 


Suppose  k=0.  Then 


fn-2 

n-1  roo  -O/xl  -D/x  ir-1 

GKn^m^O)  =  -  /  e  [l-e  J  - 

2n  •'0*0  X 


-D/x 

e  do 


(n-1)(m-1)  n-2  i  /m-2\  1  /•  oo  -D(i+2>/x 

-  Z  (  )  -  /  ® 

2n  '  i=C  \  i  /  X  ^0=0 


(n-1)(m-1)  m-2  i  /m-2\  1 

-  E  <-'«>  (  )  — 

2n  i=0  Vi/  i+2 


n-1  m  i  /m\ 

—  E  <-i>  (  V^- 

2mn  1=2  \i/ 


1) 


m 

n-1  d  <1-z)  -1 
2mn  dz  z 


z=1 


Glln^m^O)  = 


n-1 


?mn 


(5.24) 


Unlike  GE#  G1  increases  with  n.  It  has  a  finite  upper 
bound  of  1/2m  as  sentences  grow  very  large.  Figure  5.19 
illustrates  a  family  of  GI  curves.  For  general  k/  equations 
5.23  and  5.14  yield 


n-1  r  00  -0/x  /m-2 

GKn/m^k)  =  -  /  e  (m-1)( 

2n  •'0  =  0  \  k 

m-k-2  i  /ni'k-2\  1  -(i  +  k  +  1)0/x 

x;  (-1)  f  -  e  do 

i  =  0  \  i  /  X 


From  the  derivation  following  5,21  leading  to  5.22  we  have 

(k+1)(n-1) 

Gl(n^irrk)  =  -  =  (k+1  )G  I  <  n^m,  0)  (5.25) 

2mn 


Figure  5.20  shows  GI  versus  m  for  various  k  with  n=16. 


assaiks  on  tt!£  b£ha)^i2i:  s.t  LaiaS  !!! 


GEln/m^O)  and  GIln/m/0)  each  have  limitino  values  of 
zero  as  m  approaches  infinity.  D(m^D)  is  unbounded  however. 
One  must  allow  k  to  grow  large  in  order  to  bound  D.  Suppose 
we  let  k=m-j  (j>1).  Then 


m-1  1 

limit  £(mrk)  =  limit  ^  -  =  0 

m— *00  m-*oo  i=m-j  +  1  i 


limit  GE(n#m^k) 
•  —♦00 


m- j  +  2 
limit  —  - 
m-*oo  mn 


^  A 

i  =  0  i+2 


1 

n 


i=C 


1 

i+2 


<m- j+2> (n-1 >  n-1 

limit  Gl(n/m^k)  =  limit  ■  =  — • 

•  -♦00  •-♦00  ?mn  2n 
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GKn.m.O)  (%) 


7- 


Figur*  5.19.  Gap  probability  (Matbod  I)  «t.  umpla  tiza. 
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This  approach  puts  at  zero  but  forces  GE  and  GI  outside 
the  acceptable  region.  Suppose  we  let  m  and  k  approach 
infinity  such  that  k/m  approaches  the  value  a  (0<a<1).  Then 


k+1 

n-2 

1 

a  n-2 

limit 

GE  (n^ 

m^k)  =  limit 

E 

— 

=  -  E 

m— * 

m— * 

mn 

i  =  0 

i+2 

n  i  =0 

(k+1) (n-1) 

a(n-1) 

limit 

Glln, 

m/k)  =  limit 

= 

m  — * 

m— * 

?mn 

?p 

m-1 

1 

limit 

0 (m/ k ) 

=  1  i  m  i  t  E 

- 

m— * 

m  — *  i  =  k  + 1 

i 

An  expression  for  this  last  limit  must  be  found  in  oraer  to 
aralytically  determine  the  optimum  value  of  a  (based  on  the 
quality  function  defined  earlier).  We  begin  by  recognizing 
the  following  inequality 

n  1  r  m-1  1  m-1  1 

E  -  <  /  -  <  E  - 

i=k  i  t  i=k-1  i 


Now  notice  that 


-A 


log(m)-log(k) 


Icglm/k) 


Takinq  the  limit  as  m  approaches  infinity  we  see  that 


m 

1 

1 

1  m-1 

1 

m-1 

1 

limit 

E 

i=k 

- 

=  limit 

-  - 

—  *  E 

- 

=  limit 

E 

- 

m-*oo 

« 

m-»oo 

m 

k-1  i=k-l 

i 

m-V 

i=k-1 

i 

Hence 

it 

must 

be  that 

m-1  1  m-1 

limit  E  “  “  limit  log  —  =  logd/a)  =  -log(a) 

m-+oo  i=k  +  1  i  m— *oo  k  +  1 
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We  wish  to  Biniiiize  the  noraelized  distance  froa  the  origin; 
i  .t.t 


1  /2 


The  square  root  may  be  dropped  without  affecting  the  minimi¬ 
zation.  hotice  that  the  limit  of  GE  and  GI  may  be  written 
as  a  (different)  constant  (actually  a  function  of  n)  tiroes 
the  parameter  a.  Therefore^  the  form  of  the  function  to  be 
minimized  is 


/- 1  o  Q  ( a ) 


2 


Setting  the  derivative  (with  respect  to  a)  equal  to  zero  we 
have 


0 


2  /q  \ 

a  =  -( — )  log(a)  (5.26) 

VdK/ 


Checking  the  value  of  the  second  derivative  at  the  value  of 
a  thus  obtained  we  have 

u,<.,  *  .0  g  *  zQ 


Using  the  value  in  equation  5.26  we  obtain 


202 


2 


1 


for  0<a<1 


2 


2 


-  >  0 

log(a) 


Therefore  the  solution  is  indeed  a  minimum.  Numerical  solu¬ 
tion  of  5.26  (for  n=16)  yields  a  =  . 081)1  for  method  E  and 
a=.0546  for  method  I.  This  gives  corresponding  optimum 
points  of  GE=.0119  at  0=2. 52x  and  GI=.0256  at  D=2.91x. 
Since  one  would  be  required  to  remember  the  k+1  largest 
values  (and  the  minimum  value)#  the  process  which  allows  m 
and  k  to  grow  arbitrarily  large  is  computationally  infeasi¬ 
ble.  The  knowledge  of  our  proximity  to  the  optimum  is  how¬ 
ever  quite  important. 

In  Figure  5.21  we  plot  GE  versus  D  for  k=l)  and  various 
n.  The  choice  of  m  is  quite  critical.  For  the  case  of  n=32 
it  can  be  shown  that  the  acceptable  range  for  m  is  approxi¬ 
mately  16  to  24.  If  m  is  smaller  GE  is  too  large;  if  m  is 
larger  then  D  is  too  large.  (The  larger  region  of  accepta¬ 
bility  for  GI  causes  a  somewhat  wider  choice  of  acceptable 
m . ) 

In  order  to  examine  the  behavior  of  different  values  of 
k#  we  fix  n  (at  16)  and  plot  GE  versus  0  in  Figure  5.22. 
Notice  that  performance  increases  (i.e.#  normalized  distance 
from  the  origin  decreases)  as  k  increases  (with  correspond¬ 
ing  increase  in  m).  That  is#  for  those  curves  plotted#  for 
any  m#k  pair  there  exists  m*  such  that  the  performance  at 
exceeds  the  performance  at  m#k.  Also#  for  each  m 
there  is  an  optimum  value  of  k  (which  maximizes  the  quality 
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function).  Figure  5.23  shows  the  value  of  the  optiaial  k  for 
each  m<200.  The  average  slope  of  the  functions  approx inat es 
the  value  of  the  parameter  a.  Figure  5.24  gives  the  perfor¬ 
mance  for  m<200  at  the  optimum  k  for  each  m.  Also  shown  is 
the  limiting  performance.  The  performance  approaches  the 
limit  quickly  at  first  and  therefore  one  may  achieve  perfor¬ 
mance  quite  close  to  the  optimum  for  small  m  (and  therefore 
small  k).  To  see  this  more  clearly  we  have  plotted  in  Fig¬ 
ure  5.25  the  fraction  by  which  the  performance  at  each 
m^k-optimal  point  exceeds  the  limit'ng  performance.  That 
is#  if  L(Q)  is  the  limiting  performance#  then  CQ(m#k-opt)- 
L(Q)3/L((9)  is  the  ratio  plotted  versus  m  in  the  figure. 
Notice  that  convergence  to  the  limit  is  faster  than  a  nega¬ 
tive  exponential  (which  would  appear  as  a  straight  line  in 
this  figure). 

A  rule  of  thumb  for  choosing  m  and  k  is  as  follows. 
Choose  m>40  and  k  approximately  ”  percent  of  m  for  method  E 
and  5.5  percent  for  method  I.  We  see  in  Figure  5.25  that 
using  such  a  rule  for  the  exponential  distribution  of  delay# 
n=16  and  the  given  regions  of  acceptability  achieves  perfor¬ 
mance  within  5  percent  of  the  limiting  performance.  In  sub¬ 
sequent  sections  we  relax  these  assumptions  and  find  that 
the  rule  continues  to  provide  good  performance. 
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GE(16,m,k)  (%) 


Figura  5.22.  Gap  probability  (Mathod  E)  v(.  mean  range. 
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Figure  5.23.  Optimal  k  for  given  m. 
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Q(m,k-opt)  •  L(0) 
L<Q) 


5. 3. 3. 6. 2  lbs  £&c&QfiDiiai  disi£ibuii&D 


In  section  5. 3. 3. 1.3  we  noted  that  the  shifted  exponen¬ 
tial  (and  the  shifted  Erlangian)  could  sometimes  be  used  to 
approximate  an  observed  delay  distribution.  Analytic 
results  for  this  distribution  are  shown  in  this  section  to 
be  similar  to  the  results  for  the  exponential  distribution. 

The  key  observation  is  that  equations  5.6^  5.7^  5.4  and 
5.5  are  invariant  under  a  linear  shift  (translation  of 
axes).  For  example 


RID  I  m) 


00 

CS(t+0)-S(t)D 


yt-b=a 


m- 1 


CS (t-b+D)-S(t-b): 


dS(t-b) 


Th-'  resulting  formulas  are  therefore  identical  to  those 
for  the  exponential  distribution  with  one  exception.  The 
mean  value  x  must  be  replaced  with  the  parameter  w  =  x-b. 
This  appears  in  the  formula  for  D(m^O)  (i.e./^  equation 
5.13).  Let  b(m»k»b)  be  the  mean  range  of  m  samples  with  the 
highest  k  discarded^  given  the  delay  is  distributed  with 
exponential  distribution  shifted  by  b.  Then 
x-b 

D(m/k/b)  =  -  0(m^k/Q)  (5.27) 

X 


Therefore  as  b  increases  O(m^k^b)  decreases.  As  I  ap¬ 
proaches  X  the  limiting  range  is  teror  which  is  expected 
since  the  limiting  distribution  is  that  of  a  constant  valued 


random  variable.  The  results  of  the  previous  section  there¬ 


fore  apply  here  after  an  appropriate  scaling  of  the  D  values 
by  the  factor  <x-b)/x. 

5. 3. 3. 6. 3  Ihe  &rjLaQgi2Q  faiiiy  gi  diilcibUtiQDS 

Section  5. 3. 3. 1.3  shows  that  the  Frlangian  (and  shifted 
Erlangian)  family  of  distributions  provide  useful  approxima¬ 
tions  to  observed  distributions.  In  this  section  results 
for  a  family  of  Erlangian  distributions  appear.  The  family 
to  be  considered  appears  in  Figure  5.26  as  a  family  of  pro¬ 
bability  density  functions-  Symbolic  integration  of  equa¬ 
tions  5.6^  5.7^  5  .A  and  5.5  becomes  exceedingly  complicated 
for  an  Erlangian  distribution.  Therefore  we  resort  to  a 
numerical  integration  technique  in  order  to  solve  for  6F 
and  SI. 

Using  the  so  called  parabolic  rule  of  numerical  in¬ 
tegration  IHi  Id  7A3  (p.  96)/  we  evaluate  equations  5.7/  5.4/ 
and  5.5.  The  results  appear  in  Figure  5.27  as  a  set  of 
points  in  the  D-GE  plane.  We  have  fixed  n  at  16  and  show 
the  values  obtained  for  r=5/  10/  and  50/  with  m=2/  10/  20/ 
30/  40/  50/  and  100/  and  k=0/  1/  2/  3/  4/  5/  and  10  (and 
k<m-1 ) . 

There  many  similarities  betwv“en  these  results  and  those 
for  the  exponential  distribution  (i.e./  r=1 ) .  Connecting 
the  points  of  constant  k  we  see  that  the  hyperbolic  shape 
persists.  We  also  notice  that  the  distance  between  constant 
k  curves  shrinks  rapidly  with  increasing  k/  as  before. 
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Notice  that  6E  (and  GI^  not  shown)  decreases  much  more  ra¬ 
pidly^  as  £  increases/  than  for  the  r~1  case.  (In  fact  a 
different  scale  was  required  to  s^ow  some  degree  of  detail.) 
This  is  not  unexpected;  as  mentioned  earlier  the  r=1  case 
was  expected  to  provide  a  lower  bound  on  performance.  It  is 
expected  that  as  variability  of  delay  decreases  less  buffer¬ 
ing  will  be  necessary  to  provide  the  same  degree  of  gap  pro¬ 
bability.  This  is  indeed  the  case  and  at  the  limit  as  r 
approaches  infinity  delay  becomes  constant  and  buffering  is 
no  longer  required  to  reduce  the  gap  probability. 

Relative  distance  from  the  origin  (i.e./  communication 
quality)  is  based  on  the  region  of  acceptability  which  we 
have  not  specified  for  this  family  of  distributions.  We 
have  circled  the  points  m  =  40/  5C/  and  100  with  V  -  3/  4/ 
and  10  respectively  (i.e./  those  closest  to  the  rule  of 
thumb).  Notice  that  points  provide  a  rather  good  balance  of 
mean  delay  with  gap  probability. 
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Figure  5.26.  Erlang  family  of  deniity  function*. 
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p(in,k)/x 


5. 3. 3. 7  Slgyisilgn  C£SUil& 

Some  assumptions  were  made  in  earlier  sections  in  order 
to  provide  a  tractable  model  of  the  system.  Even  with  these 
assumptions^  analytic  results  were  obtained  only  for  a  smalt 
class  of  distributions.  We  therefore  have  resorted  to  simu¬ 
lation  in  order  to  relax  some  of  the  assumptions  made. 
Presented  in  this  section  are  the  results  of  that  simula¬ 
tion. 

The  simulation  program  is  driven  by  a  sequence  of  de¬ 
lays.  Therefore  it  is  possible  to  remove  the  assumption  of 
statistical  independence  among  samples.  This  assumption  is 
relaxed  by  using  actual  delay  strings  from  ARPANET  measure¬ 
ments  and  from  simulation. 

It  is  also  possible  to  remove  the  assumption  of  fixed 
sentence  size.  Suggested  by  the  early  work  of  Norwine  and 
Murphy  CNcrw  383^  we  have  used  a  geometric  distribution  of 
sentence  length  (in  terms  of  packets)/  with  a  mean  which 
corresponds  rouahly  to  the  4.14  seconds  reported  in 
tNorw  383  as  the  average  length  of  a  talkspurt. 

The  simulations  were  performed  as  follows.  First  the 
delay  string  is  "randomly"  divided  into  sentences.  At  the 
beginning  of  each  sentence  a  selection  of  D  is  made  bas'^d  on 
the  particular  algorithm  'range  monitor  or  delay  tracker) 
being  simulated.  knowing  the  delay  of  the  first  message  of 
the  sentence  and  r  ue  determine  the  number  of  gaps  which 
would  have  occurred  were  this  the  sequence  of  delays  experi¬ 
enced  by  an  actual  sentence.  This  is  repeated  for  each 
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sentence  In  the  string  and  the  results  are  recorded 


We  show  only*  the  results  using  two  of  the  measured 
delay  strings.  D  is  now  expressed  in  msec  instead  of  a  mul¬ 
tiple  of  the  mean  delay.  In  Figures  5.28(a)  and  5.29(a)  are 
the  results  of  the  range  monitoring  techniques  using  e  =  2f 
I,  U,  5/  6,  7,  8,  9,  10/  12/  14/  16/  18/  20/  25/  30/  35/  40/ 
45/  50/  60/  70/  80/  90/  and  100/  with  k  =  0/  1/  2/  3/  4/  5/ 
and  ID  (k<m-1).  The  performance  retains  the  hyperbolic 
character  to  a  large  extent.  Notice  that  a  wide  range  of 
m/k  pairs  give  performance  below  GE-.03  (and  GI=.06/  not 
shown).  The  m>4Q/  k=.03m  rule  provides  good  performance 
here  as  well/  though  not  apparent  from  the  figure. 

In  Figures  5.28(b)  and  5.2°(b)  are  the  results  of  the 
delay  tracker  methods.  The  constant  c  scheme  (represented 
by  x's  in  the  figures/  each  x  is  the  result  of  a  single 
value  or  c)  performs  quite  well  considering  its  simplicity. 
One  must/  however/  be  quite  careful  in  the  selection  of  c. 
Note  that  the  range  of  "good"  c  varies  with  delay  distribu¬ 
tion/  which  is  effected  by  a  number  of  factors  of  which  the 
user  is  unaware  (i.e./  network  distance/  network  load/ 


etc.) 


Parcant  packats  causing  gaps  (Mathod  E) 


Percent  peckets  causing  gaps 


D  (msec) 

Figure  S.29(a).  Range  prediction  performance  simulation  (Method  E,  measurement,  10  hope) 
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Percent  packets  causing  gaps 
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A  very  interesting  fact  is  now  apparent  regarding  the 
performance  of  the  scheme  which  dynamically  selects  c.  Each 
point  (a  +  sign  in  the  figures)  shows  the  performance  for  a 
particular  initial  value  of  c.  Larger  values  of  c  may  cause 
both  larger  mean  delay  and  more  gaps!  While  the  performance 
is  quite  good  in  terms  of  gap  probability^  0  depends  criti¬ 
cally  on  the  selection  of  the  initial  value  of  c.  This  ins¬ 
tability  is  effected  by  the  selection  of  the  gapless  string 
length  parameter  s.  It  would  probably  be  possible  to  find 
an  s  which  would  result  in  stable  behavior  for  these  partic¬ 
ular  delay  strings.  Finding  an  s  which  guarantees  stability 
for  all  (or  even  some  most  likely)  strings^  however/  is 
probably  impossible. 

5.4  C2Q£iusiQQ£ 

Regarding  source  buffering  for  stream  traffic  communi¬ 
cation/  we  found  that  no  packet-by-packet  strategy  could 
adapt  to  the  rapidly  changing  network  delay  in  a  way  which 
effectively  reduced  end-to-end  delay  (or  its  variance)  or 
significantly  increased  throughput.  Therefore/  we  conclude 
that  sending  strategy  should  remain  fixed  for  long  periods 
and  only  adapt  to  major  changes  in  network  performance/ 
perhaps  by  using  feedback  from  the  destination  buffering 
algorithm . 

In  the  area  of  destination  buffering/  the  results  are 
more  promising.  One  can  devise  buffering  schemes  which 


effectively  balance  the  frequency  (and  duration)  of  output 


gaps  against  end-to-end  delay.  We  have  produced  a  franework 
by  which  such  schemes  may  be  compared.  Ry  applying  that 
frameworks  we  have  founds  for  examples  that  among  these 
techniquess  delay  range  prediction  is  advantageous  to  delay 
tracking.  This  is  the  case  because  delay  tracking  requires 
user  input  and  may  be  unstable  in  certain  cases.  Our  ap¬ 
proach  of  looking  at  each  end  of  a  continuum  of  playback 
schemes  has  produced  bounds  of  performance  within  which  the 
performance  of  all  schemes  in  the  continuum  must  lie.  We 
have  gained  in  understanding  the  tradeoff  between  gaps  and 
delay.  The  above  should  provide  some  useful  tools  for 
designing  new  and  better  buffering  and  playback  schemes. 

In  a  more  general  sense  we  have  gained  some  insight 
into  the  area  of  network  delay  distribution  and  the  indepen¬ 
dence  of  successive  delay. 
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CHAPTER  6 


CONCLUSIONS  AND  SUGGESTIONS  FOR  FUTURE  RESEARCH 


6-1  CQQCiySiQQJ 

»(e  have  identified  three  areas  within  the  ARPANET  cro- 
cedures  in  which  the  performance  of  stream  traffic  communi¬ 
cation  may  be  significantly  enhanced.  In  each  case  it  was 
demonstrated  (by  simulation)  that  the  traditional  forms  of 
data  communication  (i.e./  the  LD  and  HT  classes  defined  in 
Chapter  1)  would  have  improved  performance  as  well.  The 
underlying  cause  of  each  of  the  areas  of  poor  performance 
was  eventually  found  to  be  the  routing  update  procedure. 
However#  the  lessons  learned  are  worth  separate  mention.  We 
now  state  three  rules  of  thumb#  derived  from  this  research# 
which  apply  in  general  to  packet  switching  (at  least  of  the 
ARPANET  variety). 

1)  Routine  loops  must  be  controlled. 

Z)  Great  care  must  be  taken  whenever  a  locally  optimum 
policy  is  violated#  in  order  to  insure  that  the  vio¬ 
lation  improves  global  performance  in  some  signifi¬ 
cant  way. 

3)  In  (at  least)  large  networks#  periodic  update  rout¬ 
ing  procedures  require  excessive  overhead#  There¬ 
fore#  for  large  networks  we  must  update  routing 
information  by  schemes  which  are  asynchronously 

2?3 

.  “  _ _ _ 


triggered  by  the  failure  or  repair  of  network  com¬ 
ponents  or  by  heavy  congestion  in  the  network. 

In  Chapter  2  we  found  that  the  most  effective  periodic 
update  routing  procedure  (of  those  considered)  was  the  local 
loop-free  scheme  LLFR.  The  general  loop-free  scheme  LFR  was 
less  effective  because  of  the  infrequent  occurrence  of  mul¬ 
tinode  loops  coupled  with  the  increased  overhead  required  to 
prevent  multinode  loops.  Mowever/^  it  should  be  clear  that 
under  an  asynchronous  update  procedure  we  must  require  each 
update  to  produce  a  loop-free  set  of  routing  tables.  There¬ 
fore  LFR  is  a  candidate  for  a  distributed  asynchronous  up¬ 
date  routing  procedure  in  large  networks.  We  could  call 
this  new  technique  ALFR  (pronounced  as  a  New  Englander  pro¬ 
nounces  the  first  letter  of  the  of  the  Greek  alphabet). 

Rule  2  is  generated  by  our  discovery/  in  Chapter  3/ 
that  processing  routing  packets  at  low  priority  produced 
both  local  and  global  performance  improvement  in  terms  of 
average  delay.  Under  an  asynchronous  update  scheme  this 
discovery  may  not  have  been  made.  Perhaps  under  such  a 
scheme/  since  updates  are  triggered  by  something  very  "bad" 
(or  good)/  high  priority  routing  may  produce  better  perfor¬ 
mance  globally  even  though  it  would  still  be  worse  locally. 

We  found  in  Chapter  4  that  periodic  update  routing  is 
costly  in  large  networks.  With  periodic  update  routing/  in 
order  to  insure  that  news  of  a  failure  is  propagated  through 
a  large  network  it  is  necessary  to  reduce  the  period  of  the 
routing  updates.  As  the  network  grows  bigger  so  does  the 
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overhead  due  to  routing.  With  such  high  overhead  the  pur¬ 
pose  of  adaptive  routing  (i.e.^  to  reduce  average  delay  and 
increase  throughput  in  the  face  of  changing  topology  or 
traffic  patterns)  is  defeated. 

Our  investigation  of  sending  and  receiving  strategies 
has  yielded  the  following  results.  We  found  no  effective 
method  of  using  delay  prediction  on  a  message-by-message 
basis  to  assist  in  the  sending  strategy  due  to  the  large 
variability  of  delay  for  successive  messages.  Therefore^  we 
conclude  that  the  sending  strategy  should  remain  fixed  for 
some  "long"  periods  of  time  and  modified  only  as  a  result  of 
large  variations  in  network  performance.  The  study  of  des¬ 
tination  buffering  proved  to  be  more  fruitful.  The  delay 
tracking  methods  are  attractive  because  of  implementation 
ease/  but  the  requirement  that  the  user  specify  a  starting 
value  is  undesirable.  For  "good"  startin.,  values  the  delay 
trackers  perform  as  well  as  the  slightly  more  complicated 
range  predictors.  Unfortunately/  however/  a  good  selection 
of  starting  value  is  dependent  on  factors  of  which  the  user 
is  completely  unaware.  On  the  other  hand/  using  a  fixed  (or 
automatically  adjusted)  m-k  pair  allows  the  automatic  selec¬ 
tion  of  "destination  wait  time"  based  more  on  the  network 
dynamics.  Therefore  range  prediction  is  preferable  to  delay 
t  racking. 
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6.2  &U9afiSliQDS  t9£  iu£u£s  iQvsniaaiistn 

Several  areas  related  to  this  work  have  been  Left  un¬ 
touched  here.  We  have  considered  only  part  of  the  larqer 
problem  of  network  design.  That  is^  to  optimize  the  design 
(e.g.«  minimize  the  cost)  of  a  network  which  is  to  carry 
several  classes  of  traffic^  each  with  a  different  set  of 
constraints#  over  the  set  of  design  variables  which  includes 
topology#  capacity  assignment#  routing  procedure#  system 
priorities#  buffering#  and  flow  and  congestion  control. 
Some  of  the  more  fruitful  individual  areas  would  seem  to  be 
routing  and  flow  and  congestion  control.  ‘le  stated  earlier 
that  asynchronous  update  routing  procedures  deserve  investi¬ 
gation.  Some  work  has  been  done  by  Kleinrock  and  Fultz 
CFult  723  in  this  area  for  small  networks.  The  topic  should 
be  further  studied  especially  with  regard  to  large  networks. 

In  Chapter  5  we  modeled  the  stream  traffic  delay  as  an 
independent  random  variable  possessing  some  probability  dis¬ 
tribution  function.  A  better  characterization  using  random 
processes  would  allow  the  creation  of  more  accurate  models 
of  the  destination  buffering  problem  as  well  as  aid  in  the 
above  design  problem.  The  buffering  problem  itself  could 
also  be  extended  to  include  a  notion  of  gap  duration.  The 
model  shoulo  also  be  used  or  extended  to  provide  information 
regarding  the  minimum  buffer  size  required  (at  the  destina¬ 
tion  device)  to  guarantee  certain  performance. 


Finally  the  buffering  analysis  could  be  extended  im¬ 
mediately  to  the  other  packet  technologies  mentioned  in 


Chapter  1  by  providing  a  delay  distribution  in  each  case. 
Also^  the  destination  buffering  and  associated  delay  moni¬ 
toring  could  be  used  to  provide  feedback  to  the  source  in 
order  that  changes  be  made  in  sending  strategy  and  the 
compression  scheme^  at  least  in  the  case  of  speech. 
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